Re: [Neo4j] How to query based on properties

2011-02-24 Thread Craig Taverner
Since your values are discrete, Davids solution would be a good one.
Connecting each node to a node representing the value of the property. So if
each node has 5 properties (in your example below), there would be five
relationships to value nodes, each of which would in turn be related to
their property node.

student --IN_YEAR-- '2nd year' -- VALUE -- years

This is like having a bunch of category indexes that you can use to find all
students with a particular property/value pair. But for your combined
property/value search, I think you would still end up traversing through
quite a lot of the graph to get your answer.

So there is an elaboration on this which takes it one step further. Create
nodes that represent the combinations of values. If you have 10 properties,
each with on average 5 possible values, you have a maximum of 5^10
theoretically possible values. The real maximum number of combinations will
also never be more than the number of students. If it does reach the number
of students, there are no students in common, so that is unlikely.

If each student is connected instead to one of these nodes, then finding
students in common is a trivial depth 2 traversal.

However, the complexity is now moved to the step for creating the graph. As
you load students into the graph, you need a fast way to see if that
students combination of properties and values already exists, and if so,
link it, and if not, create it and link it.

There are two solutions to this:

   - make a hash of all the values and index that in lucene, so you can find
   it later
   - use the amanz-index
https://github.com/craigtaverner/amanzi-index(still in progress).
This is a pure tree index that builds basically the
   same structure I described above.

The lucene approach is better known, since Neo4j has had lucene for years.
My index is still a prototype.

On Thu, Feb 24, 2011 at 8:54 AM, agam...@gmail.com wrote:

 Hi David

 I was thinking on these lines my self, but was unable to formulate it. I
 think Ill elaborate on the actual problem as you've suggested.

 There are a  number of college students who I have gathered various
 information about, example:

 1. What their major is (4 options)
 2. What year they are in (4 options)
 3. Favourite genre of music and movies (4 options each)
 4. A few yes/no questions
 5. I have a list of who's friends with who in this sample

 Now I want to see the people belonging to Person A's  most populated common
 property set.
 Assuming that number is 5properties out of 10, I next want to see for 4
 properties (which may be different, but obviously for the same5-1 as well).

 I hope this makes it clearer.

 Thanks!
 Sent on my BlackBerry® from Vodafone

 -Original Message-
 From: David Montag david.mon...@neotechnology.com
 Sender: user-boun...@lists.neo4j.org
 Date: Wed, 23 Feb 2011 23:30:11
 To: Neo4j user discussionsuser@lists.neo4j.org
 Reply-To: Neo4j user discussions user@lists.neo4j.org
 Subject: Re: [Neo4j] How to query based on properties

 Agam,

 Depending on the set of possible values, you could represent the properties
 with relationships instead. A unique property value can then be represented
 by a node, which would be linked to all nodes that have that value. The
 relationship type could indicate the property. The value nodes would then
 be indexed so that you can find the right node when setting the property
 (i.e. creating a relationship to the value node).

 Also, it would be great if you could elaborate a bit more on the actual use
 case behind this algorithm. That way, a more suitable solution might
 emerge,
 solving your problem in a different way.

 Thanks,
 David

 On Wed, Feb 23, 2011 at 10:36 PM, Agam Dua agam...@gmail.com wrote:

  Hey
 
  I'm a graph database and Neo4j newbie and I'm in a bit of a fix:
 
  *Problem Description*
  Let's say I have 'n' nodes in the graph, representing the same type of
  object. They have certain undirected links between them.
  Now each of these 'n' nodes has the same 10 properties, the *values* of
  which may differ.
 
  *Problem Statement*
  Take starting node A. I need to find a way to traverse all the nodes of
 the
  graph and print out which nodes have the most properties in common with
 A.
  For example, if A, C, D, E, F, G have 'x' properties in common I want to
  print the nodes.
  Then, I want to print the nodes which have 'x-1' properties with the same
  value. Then 'x-2', and so on.
 
  *Question*
  Now my question is, is this possible? If so, what would be the best way
 to
  go about it?
 
  Thanks in advance!
  Agam.
  *
  *
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 



 --
 David Montag
 Neo Technology, www.neotechnology.com
 Cell: 650.556.4411
 david.mon...@neotechnology.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user
 

Re: [Neo4j] [Blog Post] Using graph databases for knowledge representation and reasoning.

2011-02-24 Thread Alfredas Chmieliauskas
Great post.

Despite that the examples you give can be replicated with rdf and
spraql (forget owl). One might argue that OWL is a good example of
over-engineering, as in practice only the owl:sameAs is used :-)
But I think the real power of gremlin is within the grateful dead and
spreading activation example. This is a radically different way that
can be also used for reasoning and inference.
I mean that example really has to get more publicity as its something
completely different and useful :-)

Afredas


On Thu, Feb 24, 2011 at 8:05 AM, Peter Neubauer
peter.neuba...@neotechnology.com wrote:
 Very nice post Marko, clean and to the point!

 Cheers,

 /peter neubauer

 GTalk:      neubauer.peter
 Skype       peter.neubauer
 Phone       +46 704 106975
 LinkedIn   http://www.linkedin.com/in/neubauer
 Twitter      http://twitter.com/peterneubauer

 http://www.neo4j.org               - Your high performance graph database.
 http://startupbootcamp.org/    - Öresund - Innovation happens HERE.
 http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.



 On Thu, Feb 24, 2011 at 12:27 AM, Marko Rodriguez okramma...@gmail.com 
 wrote:
 Hi,

 I wrote a blog post regarding knowledge representation and reasoning using 
 graph databases:
        
 http://markorodriguez.com/2011/02/23/knowledge-representation-and-reasoning-with-graph-databases/

 Automated reasoning is currently dominated by RDF and its stack of 
 technologies. However, the standard meta modeling language OWL is restricted 
 to description logics. While this is all fine and good, it must be said that 
 there are numerous types of logics. I think that because graph databases 
 don't have a such strict layered cake, the graph database scene is ripe to 
 be a foundation for the exploration and exploitation of other reasoning 
 structures and algorithms.

 Thats that,
 Marko.

 http://markorodriguez.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user

___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Maven clean install error The source must not be a directory

2011-02-24 Thread Peter Neubauer
Brendan,
no this is not the cause but recovery tests that on purpose crash the
recovery process.

Could you try to exclude the tests with

mvn clean install -Dmaven.test.skip=true

And tell me if how/that works? Also, notice the git.cmd setting for
windows in https://github.com/neo4j/graphdb/blob/master/kernel/pom.xml#L127
to make sure the Neo4j version is properly resolved.

I am going to spend more cycles at getting windows builds to behave
right next week. Many of these are due to messages.log not being
closed properly on shutdown on Windows.

Cheers,

/peter neubauer

GTalk:      neubauer.peter
Skype       peter.neubauer
Phone       +46 704 106975
LinkedIn   http://www.linkedin.com/in/neubauer
Twitter      http://twitter.com/peterneubauer

http://www.neo4j.org               - Your high performance graph database.
http://startupbootcamp.org/    - Öresund - Innovation happens HERE.
http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.



On Thu, Feb 24, 2011 at 8:25 AM, Brendan Cheng ccp...@gmail.com wrote:
 With MVN clean install, there are more errors.  I attached the log file.

 I notice some tests fail because of hard coded directory reference as
 
 24-Feb-2011 15:20:55
 org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog
 doInternalRecovery
 INFO: Non clean shutdown detected on log
 [target/var/xatest\nioneo_logical.log.1]. Recovery started ...
 

 is this the problem?


 On 24 February 2011 15:00, Peter Neubauer
 peter.neuba...@neotechnology.com wrote:
 Brendan,
 I saw something similar on Windows,I don't remember what was the
 cause, but could you run mvn clean install in order to make sure
 things are cleaned up properly?

 Cheers,

 /peter neubauer

 GTalk:      neubauer.peter
 Skype       peter.neubauer
 Phone       +46 704 106975
 LinkedIn   http://www.linkedin.com/in/neubauer
 Twitter      http://twitter.com/peterneubauer

 http://www.neo4j.org               - Your high performance graph database.
 http://startupbootcamp.org/    - Öresund - Innovation happens HERE.
 http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.



 On Thu, Feb 24, 2011 at 6:03 AM, Brendan Cheng ccp...@gmail.com wrote:
 Hi,

 I tried to build from the source code with Maven but failed with The
 source must not be a directory.
 I did some search on internet found this is a unresolved bug.

 I wonder how do you overcome it?

 Cheers,

 Brendan

 --- maven-dependency-plugin:2.1:unpack-dependencies (get-sources) @ neo4j 
 ---
 Unpacking C:\neo4j\com\target\classes to
  C:\neo4j\neo4j\target\sources
   with includes null and excludes:null
 org.codehaus.plexus.archiver.ArchiverException: The source must not be
 a directory.
        at 
 org.codehaus.plexus.archiver.AbstractUnArchiver.validate(AbstractUnArchiver.java:174)
        at 
 org.codehaus.plexus.archiver.AbstractUnArchiver.extract(AbstractUnArchiver.java:107)
        at 
 org.apache.maven.plugin.dependency.AbstractDependencyMojo.unpack(AbstractDependencyMojo.java:260)
        at 
 org.apache.maven.plugin.dependency.UnpackDependenciesMojo.execute(UnpackDependenciesMojo.java:90)
        at 
 org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:107)
        at 
 org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
        at 
 org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at 
 org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at 
 org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
        at 
 org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
        at 
 org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
        at 
 org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:534)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at 
 org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
        at 
 org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
        at 
 org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
        at 
 org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
 

Re: [Neo4j] Cache sharding blog post

2011-02-24 Thread Michael Hunger
It would be nice to put this discussion in the blog comments too, so that 
outside people can benefit from that?

Cheers

Michael

Am 24.02.2011 um 11:23 schrieb Jim Webber:

 Hi Mark,
 
 A nice clear post. The choice of Router is obviously key. For the given 
 routing examples based on user or geo it should be possible to map a request 
 to a server. For many other situations it may prove much harder to determine 
 which server has a warm cache because there is no user and there is no 
 overarching idea of locality that you have with the geo example to organise 
 the graph into logical shards? Are you not left with the partitioning 
 problem in these cases?
 
 Yeah you're right of course, if there's no natural key to establish a good 
 route then you're sunk. You might as well go with sticky sessions. I think 
 this approach is a valid interim pattern for domains where there is a natural 
 key for routing.
 
 For those domains which exhibit random access to data, then it's not going to 
 be very beneficial. But then I don't believe  graph sharding will be helpful 
 either - that's just about where to host graph nodes. Random behaviour means 
 you'll normally pick the wrong place to host the node, and then some kind of 
 consistency/replication protocol kicks in to redress that wrong choice.
 
 But in answering this, I wonder if there are actually two use cases here:
 
 1. Speed - addressed by hitting caches, whether through routing or careful 
 placement of nodes on db instances.
 2. Size - for datasets way past machine limits (petabytes, exabytes), we just 
 can't replicate the whole dataset on each machine because it's impractical. 
 
 For (1) I think routing is fine and equivalent to, though simpler than, 
 sharding. For (2) routing is a smaller part of the picture (Bloom filters to 
 get requests routed to a hot replica is about is interesting), but sharding 
 (the unsolved research problem) is super important.
 
 Jim
 
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user

___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Cache sharding blog post

2011-02-24 Thread Anders Nawroth
Hi!

I just added a reference to this thread as a comment on the blog post.

/anders

___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


[Neo4j] Help with IndexWriter ramBufferSize exception

2011-02-24 Thread Pablo Pareja
Hi all,

I just got this exception:

ramBufferSize 3584.0 is too large; should be comfortably less than 2048
SEVERE:
org.apache.lucene.index.IndexWriter.setRAMBufferSizeMB(IndexWriter.java:1368)
SEVERE:
org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.instantiateWriter(LuceneBatchInserterIndex.java:209)
SEVERE:
org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.init(LuceneBatchInserterIndex.java:66)
SEVERE:
org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.index(LuceneBatchInserterIndexProvider.java:135)
SEVERE:
org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.nodeIndex(LuceneBatchInserterIndexProvider.java:102)
SEVERE:
com.era7.bioinfo.bio4j.programs.ImportGeneOntology.main(ImportGeneOntology.java:107)
SEVERE:
com.era7.bioinfo.bio4j.programs.ImportGeneOntology.execute(ImportGeneOntology.java:75)
SEVERE:
com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66)
SEVERE:
com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:28)

when launching my Batch insertion program once I had updated the indexation
to the new API
any ideas about what should I do?

I'm launching the program in a 68GB RAM machine with the following
batchInserter.properties:

neostore.nodestore.db.mapped_memory=10G
neostore.relationshipstore.db.mapped_memory=12G
neostore.propertystore.db.mapped_memory=12G
neostore.propertystore.db.strings.mapped_memory=10G
neostore.propertystore.db.arrays.mapped_memory=512M

I don't know how or where to change that ramBufferSize' value, which by the
way didn't specified at any point.

Thanks in advance,

Pablo


-- 
Pablo Pareja Tobes
LinkedInhttp://www.linkedin.com/in/pabloparejatobes
Twitter   http://www.twitter.com/pablopareja

http://about.me/pablopareja
http://www.ohnosequences.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


[Neo4j] New Batch Inserter Index

2011-02-24 Thread Craig Taverner
Hi,

I have found a surprising behavior in the BatchInserterIndex in the new 1.3
series. If I want to index a non-string:

// Index node
MapString, Object props = new HashMapString, Object();
props.put( key, 123L );
index.add( id, props );
index.flush();

// Search for node
IndexHitsNode result = index.get(key, 123L);
//result.size() = 0

The above code always returns an empty result set. But if instead we add the
value as a string, but still search for an integer, we get a result.

// Index node
MapString, Object props = new HashMapString, Object();
props.put( key, Long.toString(123L) );
index.add( id, props );
index.flush();

// Search for node
IndexHitsNode result = index.get(key, 123L);
//result.size() = 1

This seems rather surprising to me. I would not have been surprised if the
opposite was true, but this way round was unexpected.

Is this correct behavior? And if so, why?

Regards, Craig
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


[Neo4j] Exception in indexer framework when commiting transaction

2011-02-24 Thread Axel Morgner
Hi,

after transition to the new indexer framework, I tried to rebuild the 
index on a small test database with 11,000 nodes.

It failed with the following exception:

java.lang.IllegalArgumentException: name and value cannot both be empty
 at org.apache.lucene.document.Field.init(Field.java:398)
 at org.apache.lucene.document.Field.init(Field.java:371)
 at org.apache.lucene.document.Field.init(Field.java:350)
 at 
org.neo4j.index.impl.lucene.IndexType.instantiateField(IndexType.java:331)
 at 
org.neo4j.index.impl.lucene.IndexType$CustomType.addToDocument(IndexType.java:159)
 at 
org.neo4j.index.impl.lucene.LuceneCommand$AddCommand.perform(LuceneCommand.java:187)
 at 
org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:231)
 at 
org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:319)
 at 
org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:447)
 at 
org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:65)
 at 
org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:517)
 at 
org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:623)
 at 
org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:586)
 at 
org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:105)
 at 
org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:86)


Thu Feb 24 14:21:56 CET 2011: --- CONFIGURATION START ---
Thu Feb 24 14:21:56 CET 2011: Physical mem: 7999MB, Heap size: 5333MB
Thu Feb 24 14:21:56 CET 2011: Kernel version: Neo4j - Graph Database 
Kernel 1.2-SNAPSHOT (revision: 8112)
Thu Feb 24 14:21:56 CET 2011: Operating System: Linux; version: 
2.6.35-25-generic; arch: amd64; cpus: 8
Thu Feb 24 14:21:56 CET 2011: VM Name: OpenJDK 64-Bit Server VM
Thu Feb 24 14:21:56 CET 2011: VM Vendor: Sun Microsystems Inc.
Thu Feb 24 14:21:56 CET 2011: VM Version: 19.0-b09

I made sure that I only add non-null values to the index.

Does anyone have an idea what's going wrong?

Greetings

Axel
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] New Batch Inserter Index

2011-02-24 Thread Mattias Persson
It's not expected behaviour, it's a bug in the batch inserting code
which I introduced when meddling with ValueContext.numeric behaviour.
I'm planning to get rid of that ValueContext thing and instead, when
indexing a number, index that in lucene as a string (just as usual)
_and_ numerically (for optimized/cool number range queries). Hopefully
next milestone. But for now I'll fix this bug.

2011/2/24 Craig Taverner cr...@amanzi.com:
 Hi,

 I have found a surprising behavior in the BatchInserterIndex in the new 1.3
 series. If I want to index a non-string:

 // Index node
 MapString, Object props = new HashMapString, Object();
 props.put( key, 123L );
 index.add( id, props );
 index.flush();

 // Search for node
 IndexHitsNode result = index.get(key, 123L);
 //result.size() = 0

 The above code always returns an empty result set. But if instead we add the
 value as a string, but still search for an integer, we get a result.

 // Index node
 MapString, Object props = new HashMapString, Object();
 props.put( key, Long.toString(123L) );
 index.add( id, props );
 index.flush();

 // Search for node
 IndexHitsNode result = index.get(key, 123L);
 //result.size() = 1

 This seems rather surprising to me. I would not have been surprised if the
 opposite was true, but this way round was unexpected.

 Is this correct behavior? And if so, why?

 Regards, Craig
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Cache sharding blog post

2011-02-24 Thread Mark Harwood
But in answering this, I wonder if there are actually two use cases here

Yes, I see the use cases as the design decision points you are forced
to make at varying points in the scale of increasing data volumes:
1) 0-10s of gigabytes:
Slam in the RAM on a single server and all is plain sailing
2) Hundreds of Gigabytes
Too big to hold all in RAM on a single server but not too big to worry
about the cost of replicating the data on disk. Use the suggested
intelligent cache router to favour replica servers with a likelihood
of a pre-warmed cache for the given keys. The cost of a cache miss is
not too catastrophic ( a local disk read vs RAM access)
3) Terabytes and above
Too big for RAM, too big to store or replicate in its entirety on each
server. The cost of not finding what you are after in RAM is then
potentially very large - not just a local disk read but due to
sharding potentially a network hop and related issues of the traversal
state must now be exchanged between server processes.

Cheers
Mark
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] New Batch Inserter Index

2011-02-24 Thread Craig Taverner
Thanks, I'll leave my Long.toString(123) workaround in place for now. Let me
know when the bug is fixed, and I can slightly simplify my code again :-)

On Thu, Feb 24, 2011 at 3:02 PM, Mattias Persson
matt...@neotechnology.comwrote:

 It's not expected behaviour, it's a bug in the batch inserting code
 which I introduced when meddling with ValueContext.numeric behaviour.
 I'm planning to get rid of that ValueContext thing and instead, when
 indexing a number, index that in lucene as a string (just as usual)
 _and_ numerically (for optimized/cool number range queries). Hopefully
 next milestone. But for now I'll fix this bug.

 2011/2/24 Craig Taverner cr...@amanzi.com:
  Hi,
 
  I have found a surprising behavior in the BatchInserterIndex in the new
 1.3
  series. If I want to index a non-string:
 
  // Index node
  MapString, Object props = new HashMapString, Object();
  props.put( key, 123L );
  index.add( id, props );
  index.flush();
 
  // Search for node
  IndexHitsNode result = index.get(key, 123L);
  //result.size() = 0
 
  The above code always returns an empty result set. But if instead we add
 the
  value as a string, but still search for an integer, we get a result.
 
  // Index node
  MapString, Object props = new HashMapString, Object();
  props.put( key, Long.toString(123L) );
  index.add( id, props );
  index.flush();
 
  // Search for node
  IndexHitsNode result = index.get(key, 123L);
  //result.size() = 1
 
  This seems rather surprising to me. I would not have been surprised if
 the
  opposite was true, but this way round was unexpected.
 
  Is this correct behavior? And if so, why?
 
  Regards, Craig
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 



 --
 Mattias Persson, [matt...@neotechnology.com]
 Hacker, Neo Technology
 www.neotechnology.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user

___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Help with IndexWriter ramBufferSize exception

2011-02-24 Thread Mattias Persson
There's this hardcoded setting for batch inserter indexes that
increases the ram buffer size to increase overall performance during
batch insertion. The formula is:

double atLeast = writer.getRAMBufferSizeMB();
double heapHint =
(double)(Runtime.getRuntime().maxMemory()/(1024*1024*16));
double ramBufferSize = Math.max( atLeast, heapHint );
writer.getRAMBufferSizeMB( ramBufferSize );

It's just something which I ballparked and obviously has problems for
heaps of your size. It's a tough one to fix in your case since you
cannot parameterize that calculation. The only way would be to modify
source code and build a new jar, or wait for it to be fixed (maybe by
limiting the max amount to 2Gb as Lucene seem to suggest). Hopefully
it can be addressed soon.

2011/2/24 Pablo Pareja ppar...@era7.com:
 Hi all,

 I just got this exception:

 ramBufferSize 3584.0 is too large; should be comfortably less than 2048
 SEVERE:
 org.apache.lucene.index.IndexWriter.setRAMBufferSizeMB(IndexWriter.java:1368)
 SEVERE:
 org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.instantiateWriter(LuceneBatchInserterIndex.java:209)
 SEVERE:
 org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.init(LuceneBatchInserterIndex.java:66)
 SEVERE:
 org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.index(LuceneBatchInserterIndexProvider.java:135)
 SEVERE:
 org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.nodeIndex(LuceneBatchInserterIndexProvider.java:102)
 SEVERE:
 com.era7.bioinfo.bio4j.programs.ImportGeneOntology.main(ImportGeneOntology.java:107)
 SEVERE:
 com.era7.bioinfo.bio4j.programs.ImportGeneOntology.execute(ImportGeneOntology.java:75)
 SEVERE:
 com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66)
 SEVERE:
 com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:28)

 when launching my Batch insertion program once I had updated the indexation
 to the new API
 any ideas about what should I do?

 I'm launching the program in a 68GB RAM machine with the following
 batchInserter.properties:

 neostore.nodestore.db.mapped_memory=10G
 neostore.relationshipstore.db.mapped_memory=12G
 neostore.propertystore.db.mapped_memory=12G
 neostore.propertystore.db.strings.mapped_memory=10G
 neostore.propertystore.db.arrays.mapped_memory=512M

 I don't know how or where to change that ramBufferSize' value, which by the
 way didn't specified at any point.

 Thanks in advance,

 Pablo


 --
 Pablo Pareja Tobes
 LinkedIn    http://www.linkedin.com/in/pabloparejatobes
 Twitter       http://www.twitter.com/pablopareja

 http://about.me/pablopareja
 http://www.ohnosequences.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Cache sharding blog post

2011-02-24 Thread Jim Webber
Hey Mark,

That's a really fantastic and useful design metric. Can paraphrase it a bit and 
write it up on the Neo4j blog/my blog? 

I'll credit my source, naturally :-)

Jim

On 24 Feb 2011, at 14:08, Mark Harwood wrote:

 But in answering this, I wonder if there are actually two use cases here
 
 Yes, I see the use cases as the design decision points you are forced
 to make at varying points in the scale of increasing data volumes:
 1) 0-10s of gigabytes:
 Slam in the RAM on a single server and all is plain sailing
 2) Hundreds of Gigabytes
 Too big to hold all in RAM on a single server but not too big to worry
 about the cost of replicating the data on disk. Use the suggested
 intelligent cache router to favour replica servers with a likelihood
 of a pre-warmed cache for the given keys. The cost of a cache miss is
 not too catastrophic ( a local disk read vs RAM access)
 3) Terabytes and above
 Too big for RAM, too big to store or replicate in its entirety on each
 server. The cost of not finding what you are after in RAM is then
 potentially very large - not just a local disk read but due to
 sharding potentially a network hop and related issues of the traversal
 state must now be exchanged between server processes.
 
 Cheers
 Mark
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user

___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Exception in indexer framework when commiting transaction

2011-02-24 Thread Mattias Persson
Use the source, Luke. Consulting the lucene source I see that you're
trying to add something where both key and value are empty. Not null,
but empty strings that is. You should have checks for that as well in
your code.

2011/2/24 Axel Morgner a...@morgner.de:
 Hi,

 after transition to the new indexer framework, I tried to rebuild the
 index on a small test database with 11,000 nodes.

 It failed with the following exception:

 java.lang.IllegalArgumentException: name and value cannot both be empty
     at org.apache.lucene.document.Field.init(Field.java:398)
     at org.apache.lucene.document.Field.init(Field.java:371)
     at org.apache.lucene.document.Field.init(Field.java:350)
     at
 org.neo4j.index.impl.lucene.IndexType.instantiateField(IndexType.java:331)
     at
 org.neo4j.index.impl.lucene.IndexType$CustomType.addToDocument(IndexType.java:159)
     at
 org.neo4j.index.impl.lucene.LuceneCommand$AddCommand.perform(LuceneCommand.java:187)
     at
 org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:231)
     at
 org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:319)
     at
 org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:447)
     at
 org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:65)
     at
 org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:517)
     at
 org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:623)
     at
 org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:586)
     at
 org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:105)
     at
 org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:86)


 Thu Feb 24 14:21:56 CET 2011: --- CONFIGURATION START ---
 Thu Feb 24 14:21:56 CET 2011: Physical mem: 7999MB, Heap size: 5333MB
 Thu Feb 24 14:21:56 CET 2011: Kernel version: Neo4j - Graph Database
 Kernel 1.2-SNAPSHOT (revision: 8112)
 Thu Feb 24 14:21:56 CET 2011: Operating System: Linux; version:
 2.6.35-25-generic; arch: amd64; cpus: 8
 Thu Feb 24 14:21:56 CET 2011: VM Name: OpenJDK 64-Bit Server VM
 Thu Feb 24 14:21:56 CET 2011: VM Vendor: Sun Microsystems Inc.
 Thu Feb 24 14:21:56 CET 2011: VM Version: 19.0-b09

 I made sure that I only add non-null values to the index.

 Does anyone have an idea what's going wrong?

 Greetings

 Axel
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Help with IndexWriter ramBufferSize exception

2011-02-24 Thread Pablo Pareja
Hi Mattias,

I guess that my only option would then be limiting the max amount to 2GB as
you say,
however this will make the insertion phase way longer than what I expected
and counted on.
Just to confirm, am I then supposed to launch my java process with at most
-Xmx2G ??

I really think this is an important flaw to work on.
I look forward to seeing it fixed in the near future.

Cheers,

Pablo


On Thu, Feb 24, 2011 at 3:19 PM, Mattias Persson
matt...@neotechnology.comwrote:

 There's this hardcoded setting for batch inserter indexes that
 increases the ram buffer size to increase overall performance during
 batch insertion. The formula is:

double atLeast = writer.getRAMBufferSizeMB();
double heapHint =
 (double)(Runtime.getRuntime().maxMemory()/(1024*1024*16));
double ramBufferSize = Math.max( atLeast, heapHint );
writer.getRAMBufferSizeMB( ramBufferSize );

 It's just something which I ballparked and obviously has problems for
 heaps of your size. It's a tough one to fix in your case since you
 cannot parameterize that calculation. The only way would be to modify
 source code and build a new jar, or wait for it to be fixed (maybe by
 limiting the max amount to 2Gb as Lucene seem to suggest). Hopefully
 it can be addressed soon.

 2011/2/24 Pablo Pareja ppar...@era7.com:
  Hi all,
 
  I just got this exception:
 
  ramBufferSize 3584.0 is too large; should be comfortably less than 2048
  SEVERE:
 
 org.apache.lucene.index.IndexWriter.setRAMBufferSizeMB(IndexWriter.java:1368)
  SEVERE:
 
 org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.instantiateWriter(LuceneBatchInserterIndex.java:209)
  SEVERE:
 
 org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.init(LuceneBatchInserterIndex.java:66)
  SEVERE:
 
 org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.index(LuceneBatchInserterIndexProvider.java:135)
  SEVERE:
 
 org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.nodeIndex(LuceneBatchInserterIndexProvider.java:102)
  SEVERE:
 
 com.era7.bioinfo.bio4j.programs.ImportGeneOntology.main(ImportGeneOntology.java:107)
  SEVERE:
 
 com.era7.bioinfo.bio4j.programs.ImportGeneOntology.execute(ImportGeneOntology.java:75)
  SEVERE:
 
 com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66)
  SEVERE:
 
 com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:28)
 
  when launching my Batch insertion program once I had updated the
 indexation
  to the new API
  any ideas about what should I do?
 
  I'm launching the program in a 68GB RAM machine with the following
  batchInserter.properties:
 
  neostore.nodestore.db.mapped_memory=10G
  neostore.relationshipstore.db.mapped_memory=12G
  neostore.propertystore.db.mapped_memory=12G
  neostore.propertystore.db.strings.mapped_memory=10G
  neostore.propertystore.db.arrays.mapped_memory=512M
 
  I don't know how or where to change that ramBufferSize' value, which by
 the
  way didn't specified at any point.
 
  Thanks in advance,
 
  Pablo
 
 
  --
  Pablo Pareja Tobes
  LinkedInhttp://www.linkedin.com/in/pabloparejatobes
  Twitter   http://www.twitter.com/pablopareja
 
  http://about.me/pablopareja
  http://www.ohnosequences.com
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 



 --
 Mattias Persson, [matt...@neotechnology.com]
 Hacker, Neo Technology
 www.neotechnology.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Pablo Pareja Tobes
LinkedInhttp://www.linkedin.com/in/pabloparejatobes
Twitter   http://www.twitter.com/pablopareja

http://about.me/pablopareja
http://www.ohnosequences.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Help with IndexWriter ramBufferSize exception

2011-02-24 Thread Mattias Persson
Using the formula I supplied in my previous reply I'd say you can go
with a 32G heap or just slightly lower.

2011/2/24 Pablo Pareja ppar...@era7.com:
 Hi Mattias,

 I guess that my only option would then be limiting the max amount to 2GB as
 you say,
 however this will make the insertion phase way longer than what I expected
 and counted on.
 Just to confirm, am I then supposed to launch my java process with at most
 -Xmx2G ??

 I really think this is an important flaw to work on.
 I look forward to seeing it fixed in the near future.

 Cheers,

 Pablo


 On Thu, Feb 24, 2011 at 3:19 PM, Mattias Persson
 matt...@neotechnology.comwrote:

 There's this hardcoded setting for batch inserter indexes that
 increases the ram buffer size to increase overall performance during
 batch insertion. The formula is:

        double atLeast = writer.getRAMBufferSizeMB();
        double heapHint =
 (double)(Runtime.getRuntime().maxMemory()/(1024*1024*16));
        double ramBufferSize = Math.max( atLeast, heapHint );
        writer.getRAMBufferSizeMB( ramBufferSize );

 It's just something which I ballparked and obviously has problems for
 heaps of your size. It's a tough one to fix in your case since you
 cannot parameterize that calculation. The only way would be to modify
 source code and build a new jar, or wait for it to be fixed (maybe by
 limiting the max amount to 2Gb as Lucene seem to suggest). Hopefully
 it can be addressed soon.

 2011/2/24 Pablo Pareja ppar...@era7.com:
  Hi all,
 
  I just got this exception:
 
  ramBufferSize 3584.0 is too large; should be comfortably less than 2048
  SEVERE:
 
 org.apache.lucene.index.IndexWriter.setRAMBufferSizeMB(IndexWriter.java:1368)
  SEVERE:
 
 org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.instantiateWriter(LuceneBatchInserterIndex.java:209)
  SEVERE:
 
 org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.init(LuceneBatchInserterIndex.java:66)
  SEVERE:
 
 org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.index(LuceneBatchInserterIndexProvider.java:135)
  SEVERE:
 
 org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.nodeIndex(LuceneBatchInserterIndexProvider.java:102)
  SEVERE:
 
 com.era7.bioinfo.bio4j.programs.ImportGeneOntology.main(ImportGeneOntology.java:107)
  SEVERE:
 
 com.era7.bioinfo.bio4j.programs.ImportGeneOntology.execute(ImportGeneOntology.java:75)
  SEVERE:
 
 com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66)
  SEVERE:
 
 com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:28)
 
  when launching my Batch insertion program once I had updated the
 indexation
  to the new API
  any ideas about what should I do?
 
  I'm launching the program in a 68GB RAM machine with the following
  batchInserter.properties:
 
  neostore.nodestore.db.mapped_memory=10G
  neostore.relationshipstore.db.mapped_memory=12G
  neostore.propertystore.db.mapped_memory=12G
  neostore.propertystore.db.strings.mapped_memory=10G
  neostore.propertystore.db.arrays.mapped_memory=512M
 
  I don't know how or where to change that ramBufferSize' value, which by
 the
  way didn't specified at any point.
 
  Thanks in advance,
 
  Pablo
 
 
  --
  Pablo Pareja Tobes
  LinkedIn    http://www.linkedin.com/in/pabloparejatobes
  Twitter       http://www.twitter.com/pablopareja
 
  http://about.me/pablopareja
  http://www.ohnosequences.com
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 



 --
 Mattias Persson, [matt...@neotechnology.com]
 Hacker, Neo Technology
 www.neotechnology.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




 --
 Pablo Pareja Tobes
 LinkedIn    http://www.linkedin.com/in/pabloparejatobes
 Twitter       http://www.twitter.com/pablopareja

 http://about.me/pablopareja
 http://www.ohnosequences.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Help with IndexWriter ramBufferSize exception

2011-02-24 Thread Pablo Pareja
Cool, I thought I'd have to decrease the heap size much more.
Anyways, thanks a lot for the quick response and support.
Cheers,

Pablo

On Thu, Feb 24, 2011 at 3:44 PM, Mattias Persson
matt...@neotechnology.comwrote:

 Using the formula I supplied in my previous reply I'd say you can go
 with a 32G heap or just slightly lower.

 2011/2/24 Pablo Pareja ppar...@era7.com:
  Hi Mattias,
 
  I guess that my only option would then be limiting the max amount to 2GB
 as
  you say,
  however this will make the insertion phase way longer than what I
 expected
  and counted on.
  Just to confirm, am I then supposed to launch my java process with at
 most
  -Xmx2G ??
 
  I really think this is an important flaw to work on.
  I look forward to seeing it fixed in the near future.
 
  Cheers,
 
  Pablo
 
 
  On Thu, Feb 24, 2011 at 3:19 PM, Mattias Persson
  matt...@neotechnology.comwrote:
 
  There's this hardcoded setting for batch inserter indexes that
  increases the ram buffer size to increase overall performance during
  batch insertion. The formula is:
 
 double atLeast = writer.getRAMBufferSizeMB();
 double heapHint =
  (double)(Runtime.getRuntime().maxMemory()/(1024*1024*16));
 double ramBufferSize = Math.max( atLeast, heapHint );
 writer.getRAMBufferSizeMB( ramBufferSize );
 
  It's just something which I ballparked and obviously has problems for
  heaps of your size. It's a tough one to fix in your case since you
  cannot parameterize that calculation. The only way would be to modify
  source code and build a new jar, or wait for it to be fixed (maybe by
  limiting the max amount to 2Gb as Lucene seem to suggest). Hopefully
  it can be addressed soon.
 
  2011/2/24 Pablo Pareja ppar...@era7.com:
   Hi all,
  
   I just got this exception:
  
   ramBufferSize 3584.0 is too large; should be comfortably less than
 2048
   SEVERE:
  
 
 org.apache.lucene.index.IndexWriter.setRAMBufferSizeMB(IndexWriter.java:1368)
   SEVERE:
  
 
 org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.instantiateWriter(LuceneBatchInserterIndex.java:209)
   SEVERE:
  
 
 org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.init(LuceneBatchInserterIndex.java:66)
   SEVERE:
  
 
 org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.index(LuceneBatchInserterIndexProvider.java:135)
   SEVERE:
  
 
 org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.nodeIndex(LuceneBatchInserterIndexProvider.java:102)
   SEVERE:
  
 
 com.era7.bioinfo.bio4j.programs.ImportGeneOntology.main(ImportGeneOntology.java:107)
   SEVERE:
  
 
 com.era7.bioinfo.bio4j.programs.ImportGeneOntology.execute(ImportGeneOntology.java:75)
   SEVERE:
  
 
 com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66)
   SEVERE:
  
 
 com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:28)
  
   when launching my Batch insertion program once I had updated the
  indexation
   to the new API
   any ideas about what should I do?
  
   I'm launching the program in a 68GB RAM machine with the following
   batchInserter.properties:
  
   neostore.nodestore.db.mapped_memory=10G
   neostore.relationshipstore.db.mapped_memory=12G
   neostore.propertystore.db.mapped_memory=12G
   neostore.propertystore.db.strings.mapped_memory=10G
   neostore.propertystore.db.arrays.mapped_memory=512M
  
   I don't know how or where to change that ramBufferSize' value, which
 by
  the
   way didn't specified at any point.
  
   Thanks in advance,
  
   Pablo
  
  
   --
   Pablo Pareja Tobes
   LinkedInhttp://www.linkedin.com/in/pabloparejatobes
   Twitter   http://www.twitter.com/pablopareja
  
   http://about.me/pablopareja
   http://www.ohnosequences.com
   ___
   Neo4j mailing list
   User@lists.neo4j.org
   https://lists.neo4j.org/mailman/listinfo/user
  
 
 
 
  --
  Mattias Persson, [matt...@neotechnology.com]
  Hacker, Neo Technology
  www.neotechnology.com
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 
 
 
 
  --
  Pablo Pareja Tobes
  LinkedInhttp://www.linkedin.com/in/pabloparejatobes
  Twitter   http://www.twitter.com/pablopareja
 
  http://about.me/pablopareja
  http://www.ohnosequences.com
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 



 --
 Mattias Persson, [matt...@neotechnology.com]
 Hacker, Neo Technology
 www.neotechnology.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Pablo Pareja Tobes
LinkedInhttp://www.linkedin.com/in/pabloparejatobes
Twitter   http://www.twitter.com/pablopareja

http://about.me/pablopareja
http://www.ohnosequences.com
___
Neo4j mailing list
User@lists.neo4j.org

Re: [Neo4j] Exception in indexer framework when commiting transaction

2011-02-24 Thread Axel Morgner
Thanks Matthias,

yes, I had one empty key/value property. After adding checks, it now 
works fine.

Greetings

Axel

Am 24.02.2011 15:28, schrieb Mattias Persson:
 Use the source, Luke. Consulting the lucene source I see that you're
 trying to add something where both key and value are empty. Not null,
 but empty strings that is. You should have checks for that as well in
 your code.

 2011/2/24 Axel Morgnera...@morgner.de:
 Hi,

 after transition to the new indexer framework, I tried to rebuild the
 index on a small test database with 11,000 nodes.

 It failed with the following exception:

 java.lang.IllegalArgumentException: name and value cannot both be empty
  at org.apache.lucene.document.Field.init(Field.java:398)
  at org.apache.lucene.document.Field.init(Field.java:371)
  at org.apache.lucene.document.Field.init(Field.java:350)
  at
 org.neo4j.index.impl.lucene.IndexType.instantiateField(IndexType.java:331)
  at
 org.neo4j.index.impl.lucene.IndexType$CustomType.addToDocument(IndexType.java:159)
  at
 org.neo4j.index.impl.lucene.LuceneCommand$AddCommand.perform(LuceneCommand.java:187)
  at
 org.neo4j.index.impl.lucene.LuceneTransaction.doCommit(LuceneTransaction.java:231)
  at
 org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:319)
  at
 org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:447)
  at
 org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:65)
  at
 org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:517)
  at
 org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:623)
  at
 org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:586)
  at
 org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:105)
  at
 org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:86)


 Thu Feb 24 14:21:56 CET 2011: --- CONFIGURATION START ---
 Thu Feb 24 14:21:56 CET 2011: Physical mem: 7999MB, Heap size: 5333MB
 Thu Feb 24 14:21:56 CET 2011: Kernel version: Neo4j - Graph Database
 Kernel 1.2-SNAPSHOT (revision: 8112)
 Thu Feb 24 14:21:56 CET 2011: Operating System: Linux; version:
 2.6.35-25-generic; arch: amd64; cpus: 8
 Thu Feb 24 14:21:56 CET 2011: VM Name: OpenJDK 64-Bit Server VM
 Thu Feb 24 14:21:56 CET 2011: VM Vendor: Sun Microsystems Inc.
 Thu Feb 24 14:21:56 CET 2011: VM Version: 19.0-b09

 I made sure that I only add non-null values to the index.

 Does anyone have an idea what's going wrong?

 Greetings

 Axel
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Axel Morgner
Creative Solutions - Software Engineering
GUI  UX Design - Project Management

c/o inxire GmbH
Hanauer Landstr. 293a
60314 Frankfurt
Germany

Phone +49 151 40522060
e-maila...@morgner.de
Webhttp://www.morgner.de

___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Cache sharding blog post

2011-02-24 Thread Alex Averbuch
Cache sharding = super nice iterative/interim improvement.

It makes use of aggregate resources (RAM  CPU) across multiple servers (as
would be the case with a truly sharded Neo4j) without bothering about
partitioning (even basic consistent hashing) algorithms.

You get a viable partitioning solution by moving the logic to the
client-side. Neo4j doesn't need to worry about some of the engineering
headaches that true partitioning would bring - like performing transactions
between multiple different partitions, or load balancing partition replicas
across available servers - but clients still get some of the benefits of
true partitioning.

Like!

Some other thoughts...

As mentioned, you don't get around the difficult partitioning problem:
achieving locality in a big graph of unknown topology/access.

Also, you don't make the best use of aggregate resources. For example, if
you wanted to run one very large (read: forbidden/not-graphy) traversal like
page-rank on the entire graph you (1) would hit disk (2) would not make use
of aggregate resources (CPU,RAM). I think that's an orthogonal problem
though... regardless of how you cut your graph, the way you then traverse it
has to be resource-aware: it has to know about, and be capable of using,
the compute resources on other machines. E.g. Neo4j graph walking vs
Pregel vertex gossiping... but maybe that can be left to future
discussions.

On Thu, Feb 24, 2011 at 3:26 PM, Mark Harwood markharw...@gmail.com wrote:

  That's a really fantastic and useful design metric. Can paraphrase it a
 bit and write it up on the Neo4j blog/my blog?

 I'd be honoured.
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user

___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] [Blog Post] Using graph databases for knowledge representation and reasoning.

2011-02-24 Thread Savas Parastatidis
This is very cool. Thanks Marko!

I am a huge supporter of knowledge representation using graph
stores/databases. Here are a couple of blogs from a couple of years ago
illustrating a very simple reasoner on top of Zentity (called Famulus
back then).

http://savas.me/blog/898
http://savas.me/blog/900

I agree that traditional the first-order and description logic approaches
do not scale. I am a huge believer of probabilistic reasoning and latent
semantics.

Great stuff.

Cheers,
.savas.

On 2/23/11 3:27 PM, Marko Rodriguez okramma...@gmail.com wrote:

Hi,

I wrote a blog post regarding knowledge representation and reasoning
using graph databases:

http://markorodriguez.com/2011/02/23/knowledge-representation-and-reasonin
g-with-graph-databases/

Automated reasoning is currently dominated by RDF and its stack of
technologies. However, the standard meta modeling language OWL is
restricted to description logics. While this is all fine and good, it
must be said that there are numerous types of logics. I think that
because graph databases don't have a such strict layered cake, the
graph database scene is ripe to be a foundation for the exploration and
exploitation of other reasoning structures and algorithms.

Thats that,
Marko.

http://markorodriguez.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


[Neo4j] flocking algorithms perhaps interesting for visualization (esp. over time) in graphs

2011-02-24 Thread Michael Hunger
http://harry.me/2011/02/17/neat-algorithms---flocking/

Cheers

Michael
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] flocking algorithms perhaps interesting for visualization (esp. over time) in graphs

2011-02-24 Thread Peter Neubauer
Nice.
Added to 
http://wiki.neo4j.org/content/Visualization_options_for_graphs#JavaScript

Cheers,

/peter neubauer

GTalk:      neubauer.peter
Skype       peter.neubauer
Phone       +46 704 106975
LinkedIn   http://www.linkedin.com/in/neubauer
Twitter      http://twitter.com/peterneubauer

http://www.neo4j.org               - Your high performance graph database.
http://startupbootcamp.org/    - Öresund - Innovation happens HERE.
http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.



On Fri, Feb 25, 2011 at 1:22 AM, Michael Hunger
michael.hun...@neotechnology.com wrote:
 http://harry.me/2011/02/17/neat-algorithms---flocking/

 Cheers

 Michael
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user

___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user