Re: [Neo4j] Cypher DSL and result mapping

2011-11-04 Thread Tobias Ivarsson
I started a prototype for something similar a while back:
https://github.com/thobe/neo4j-community-experiments/tree/cypher-annotations

It has a different approach to constructing queries, it uses annotations
with query strings in them, but uses compile time annotation processing to
process those annotations in order to verify syntax. In the future (when
Cypher supports it) I'd like to do type checking as well.

What it has that is similar though is the way it converts the result to
java classes. I know that Spring-Data-Neo4j does similar things as well.
Since that kind of conversion seems to be something that multiple higher
level libraries do, it would probably make sense to move it to a shared
place.

Note please that this branch is just three hours worth of throwing some
ideas together, it's not complete, but should at least display the ideas.

-tobias

On Fri, Nov 4, 2011 at 6:10 AM, Rickard Öberg 
rickard.ob...@neotechnology.com wrote:

 Hi,

 I've briefly started to look at how to map Cypher query results into
 DTO's. I'm using the QueryDSL helper classes to do it, but instead of
 putting the projection methods into the query object (as QueryDSL does
 it), I have a separate Projection class for it. The reason is that I
 don't want to require DSL users to have access to the execution engine,
 which will obviously not be the case if you are creating queries on the
 client which are to be sent to the server.

 Here's an example of what I want it to look like when used:
 Projection projection = new Projection(engine);

 IterableFriend friends = projection.iterable( start( node( john,
 john ) )
   .match( path().from( john ).out( friend )
   .link().out( friend ).to( fof ) )
   .returns( properties( john.name, fof.name ) ),
 Friend.class );
 System.out.println( friends );
 ---
 So the Projection encapsulates the reference to the query execution,
 which means that you don't need to have a reference to it when executing
 the queries to do the projection. The iterable method takes a DSL
 query and the Java class you want the results to be projected into.
 Pretty straightforward.

 For now the code assumes that the fields in Friend is the same as the
 order of the return results. This is broken, and will be fixed later to
 use proper mapping, preferably with aliases in the return statement as
 hints.

 What are the projection methods that you would like to see? What are
 common usecases?

 /Rickard

 --
 Rickard Öberg
 Developer
 Neo Technology
 Twitter: @rickardoberg, Skype: rickardoberg
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857 (Swe); +1 650 450 3806 (US)
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Cypher DSL and result mapping

2011-11-04 Thread Tobias Ivarsson
On Fri, Nov 4, 2011 at 10:45 AM, Rickard Öberg 
rickard.ob...@neotechnology.com wrote:

 On 11/4/11 17:29 , Daniel Yokomizo wrote:
  What are the projection methods that you would like to see? What are
  common usecases?
 
  Supporting @java.beans.ConstructorProperties is nice because we can use
  immutable objects as dtos.

 Right, that is a good point, and that will definitely be supported.


+1, a really good suggestion. I'm not a big fan of that annotation per se,
it would have been nicer to annotate each parameter individually, but it is
standardized which makes it VERY compelling.

-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857 (Swe); +1 650 450 3806 (US)
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


[Neo4j] Article: The Coming SQL Collapse

2011-10-14 Thread Tobias Ivarsson
We had an interesting discussion about this internally at Neo Technology
today. We thought it might be of interest to the broader community. I don't
think the discussion is over, so it would be interesting to continue it on
the public mailing list.

It regards the initial paragraphs of an article posted to dzone recently:
http://www.dzone.com/links/rss/the_coming_sql_collapse.html
It mentions Neo4j and how the author dislikes a common way of using Neo4j
for building applications.

It would be interesting to hear suggestions on how to improve this.

Forwarded conversation follows:

On Fri, Oct 14, 2011 at 10:13 AM, Tobias Ivarsson 
tobias.ivars...@neotechnology.com wrote:

 I found this while reading feeds in bed last night:

 *The Coming SQL Collapse*
 http://www.dzone.com/links/rss/the_coming_sql_collapse.html

 (Sent from Flipboard http://flipboard.com)


 The things he say about SQL vs NOSQL is not very interesting, but I'd like
 to raise what he says about Neo4j:

  I looked at neo4j briefly the other day, and quite predictably thought
 ‘wow, this looks like a serious tinkertoy: it‘s basically a bunch of nodes
 where you just blob your attributes.‘ Worse than that, to wrap objects
 around it, you have to have them explicitly incorporate their node class,
 which is ugly, smelly, violates every law of separation of concerns and
 logical vs. physical models. On the plus side, as I started to look at it
 more, I realized that it was the perfect way to implement a backend for a
 bayesian inference engine (more on that later). Why? Because inference
 doesn‘t care particularly about all the droll requirements that are settled
 for you by SQL, and there are no real set operations to speak of.


 He attacks our pattern of building domain models with Neo4j, calling it
 ugly, smelly and in violation of every law of separation of concerns
 and logical vs. physical models. Is he right? My feeling is that he is
 brain washed with too many so called best practices, but Neo4j has been my
 main model for a long time now, my perspective is likely skewed. I'd like to
 hear your thoughts.

 --
 Tobias Ivarsson tobias.ivars...@neotechnology.com


On Fri, Oct 14, 2011 at 10:32 AM, Rickard Öberg 
rickard.ob...@neotechnology.com wrote:


 Well, I'd tend to agree with the author. Mixing persistence details with
 the domain model itself is really a bad idea. Infrastructure details should
 not pollute the domain logic as it does with the currently suggested usage
 of Neo4j. But I think both Spring Data Graph and the Qi4j usage model fixes
 this, as it allows you to keep many of those things outside of the domain
 code.

 /Rickard


On Fri, Oct 14, 2011 at 11:45 AM, Tobias Ivarsson tobias.ivarsson@
neotechnology.com wrote:

 On Fri, Oct 14, 2011 at 11:21 AM, Rickard Öberg 
 rickard.ob...@neotechnology.com wrote:

 On 10/14/11 17:16 , Tobias Ivarsson wrote:

 I was hoping for a bit more elaboration, of why it is a bad idea.

 Spring Data Neo4j operates mainly in the same way (at least it did when
 I was part of the design process), it just hides the details of it.

 The model we suggest is not to mix infrastructure details (nodes,
 relationships, traversals) with the domain logic. We suggest the domain
 logic be a separate layer, acting on domain data objects (defined as a
 set of interfaces). What we do suggest though is for those domain data
 objects to be implemented as wrappers of nodes and relationships.


 That sounds like transaction script+anemic domain model, which is an
 anti-pattern as well.


 I'm guessing the anemic domain model is what you are pointing out as the
 anti-pattern. I can see how transaction scripts are and ADM usually come
 together though.

 References:

 http://martinfowler.com/eaaCatalog/transactionScript.html
 http://martinfowler.com/bliki/AnemicDomainModel.html



 Domain logic should be in the domain objects, and so splitting them into
 several layers confuse things more than it helps.


 I agree with you, SDN solves this, and does so well.


 So is the bad part of it just the part of having the implementation of
 your domain data objects deal with the Neo4j API, instead of having DTOs
 and DAOs that you load from the database, operate on in memory, then
 store back to the db at explicit points? Or are there deeper issues than
 that?


 It relates to testability (you should be able to test domain objects
 without infrastructure), code cohesion (put domain logic in domain objects),
 and as the article author points out, separation of concerns in general.


 The testability aspect is interesting and important. I don't have enough
 SDN experience to know how it fares in that regard.

 separation of concerns in general - anything else to this? or would this
 (normally) be achieved when testability without infrastructure is achieved?


On Fri, Oct 14, 2011 at 11:49 AM, Rickard Öberg rickard.oberg@
neotechnology.com wrote:

 On 10/14/11 17:45 , Tobias Ivarsson wrote:

 I'm guessing the anemic domain

Re: [Neo4j] Clarification on read and write locks

2011-09-26 Thread Tobias Ivarsson
On Mon, Sep 26, 2011 at 2:28 AM, Bryce bryc...@gmail.com wrote:

 Is there any reason that access to the lock manager is a little difficult?
  Is there any issue with using it (both single server and HA)?  Are their
 any issues to look out for?  And does the above code look workable?


The reason the LockManager is hidden away is because it is considered an
implementation detail.
We reserve the right to change it at any point without prior notice.
From one version to another, the API of this class could potentially look
completely different, or have very different semantics.
If you write code that depend on implementation internal classes it will at
best cause you compilation errors when upgrading to a later version of
Neo4j, at worst break in subtle ways after you've deployed your application.

There is another important thing to note about your sample code:

On Mon, Sep 26, 2011 at 2:28 AM, Bryce bryc...@gmail.com wrote:

private void acquireLock( LockType lockType )
{
GraphDatabaseService graphDb = baseNode.getGraphDatabase();
if ( lockType == LockType.READ  graphDb instanceof
 AbstractGraphDatabase )
{
((AbstractGraphDatabase)
 baseNode.getGraphDatabase()).getConfig().getLockManager().getReadLock(
 baseNode );
}
else {
// default to write lock if read locks unavailable
baseNode.removeProperty( ___dummy_property_to_acquire_lock___
 );
}
}


The two different code paths in this code have very different semantics. The
first path (if it is a read lock and the db is and AbstractGraphDatabase)
will acquire a read lock and hold it indefinitely. The second path will try
to remove a dummy property, resulting in a write lock that gets tied to the
current transaction, and thus gets released when the transaction completes.

There is another class involved in the lock management in Neo4j, it is
called LockReleaser. All locks that are acquired through the public APIs are
registered with the LockReleaser. This will tie the lock to the transaction
to ensure that the lock is released when the transaction completes. If the
lock is only acquired through the LockManager, and not registered with the
LockReleaser, you need to manually release the lock.


The reason for the remove-non-existing-property-hack is that we haven't
decided on a good public API for lock management. The questions around this
are:
* Should there be release() methods for locks as well? or should locks
always be tied to the transaction?
* Where should the methods live? On the entities themselves? i.e.
node.acquireReadLock() / node.aquireWriteLock() or on the Transaction object
to signal that locks are tied to the life cycle of the transaction? i.e.
tx.readLock(node) / tx.writeLock(node)

Cheers,
-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857 (Swe); +1 650 450 3806 (US)
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Synchronization of EmbeddedReadOnlyGraphDatabase - Bug?

2011-08-01 Thread Tobias Ivarsson
I think a bit of elaboration might be in order.

EmbeddedReadOnlyGraphDatabase was created for one specific purpose:

Being able to interactively introspect a graph without having to shut down
the application that uses it.

Specifically the tools that we wanted to support with this were the Neo4j
shell and Neoclipse.

EmbeddedReadOnlyGraphDatabase (EROGD) has two major issues with way caching
is done internally in Neo4j (one issue with each cache):

   - When the EROGD reads data from the file system it will, like a normal
   EGD, cache the node and relationship objects. If a normal EGD modifies the
   graph under the feet of the EROGD, there is no way for the EROGD to know
   that the data in cache is now stale, which will lead to an inconsistent view
   of the graph. If for example the EROGD has cached Node[15] with the
   information that it is connected to some other node through
   Relationship[344], and Relationship[344] is deleted you will get
   InvalidRecordException (as you described). And of course if relationships
   are added to Node[15] these will not be seen at all by the EROGD (until
   Node[15] is evicted from the cache due to not being used for a while).
   - Neo4j also caches data on the filesystem level by memory mapping (mmap)
   hot regions of the store files. Writes to these regions will not be flushed
   to the actual file until the mmapped window is evicted due to being less hot
   than other windows, or when the transaction log for Neo4j is rotated. This
   means that from the p.o.v. of the EROGD the actual data written to disk will
   look inconsistent. Which would also lead to InvalidRecordExcaption. This
   situation is actually made even more complicated by the fact that unix
   operating systems will attempt to share memory mapped data from the same
   file between multiple processes, but the normal EGD and the EROGD will not
   make the same decisions on which regions to mmap, they might not even decide
   on the same size for mmap windows. We haven't tested how well different
   operating systems deal with reading data that was written to an mmap region
   through non-mmap syscalls from a different process, most likely this varies
   from OS to OS.

The second of these problems is of course the worst, since it cannot be
worked around. The first one can be mitigated by configuring Neo4j to not
use the object cache, by passing the cache_type=none parameter to the
constructor of the EROGD. This should really be made default for EROGD,
unless we decide to completely remove EROGD.

I hope that sheds some light on the reasons why you experience these
problems with EmbeddedReadOnlyGraphDatabase, and what the intention of
creating it was.

As a side note I can mention that I had a different idea for how to solve
the introspection-of-live-graph problem at the time
EmbeddedReadOnlyGraphDatabase was created: Create network based
implementation of the GraphDatabaseService API and connect directly to the
running instance. This would completely avoid the cache staleness problem,
but at the cost of network overhead for each graph operation, which is
probably fine for tooling purposes. With the JVM agent attach protocol it
would be possible to inject such a server into a running graph database that
wasn't originally configured for it. I in fact implemented this as the
RemoteGraphDatabase subproject.
Since my colleagues did not share my vision about that idea, this project
didn't receive much attention after its initial inception. It was also never
really used for these purposes, but rather misused for building
applications, leading us to deprecate the project. When we then later
discovered a severe bug in the implementation of the remote transaction
handling logic, we completely removed the project.
I still believe this to be a superior model for tools, but would build it
differently if I were to build it today.

-tobias

On Mon, Aug 1, 2011 at 4:48 PM, Jim Webber j...@neotechnology.com wrote:

 Hi Mathias,

 EmbeddedReadOnlyGraphDatabase is not quite what it seems, and I think
 should be deprecated/removed. The correct way for database instances to
 become consistent is through the HA protocol.

 Jim
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] cypher packages

2011-07-28 Thread Tobias Ivarsson
They should be in the neo4j-cypher-1.4.jar.

That jar-file should be among your dependencies if you simply depend on (in
maven):

dependency
  groupIdorg.neo4j/groupId
  artifactIdneo4j/artifactId
  version1.4/version
/dependency


Cheers,
Tobias

On Thu, Jul 28, 2011 at 12:07 PM, Jean-Sébastien Stoffen c-...@jsnet.bewrote:

 Hi,

 Sorry for the newbie questions who happens very frequently at the begining.

 I'm trying to submit some request to my graph using the cypher query
 language

 I get those messages :
 unable to resolve class CypherParser
 unable to resolve class ExecutionResult
 ...

 when I try the example showed here :
 http://docs.neo4j.org/chunked/1.4.M04/query-lang.html

 After some research, I've not found where I can download those packages
 I've searched for org.neo4j.cypher.javacompat but without succes.

 Could you tell me where to find it?

 Regards,
 Jean-Sébastien Stoffen.
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Django + Neo4j seamless installation error: No module named CallbackFilter

2011-07-17 Thread Tobias Ivarsson
 description, my best guess is
that the Django folks released parts of that feature too early. You'd think
there be more mentionings of this problem online then though. Do you think
you could grep through your settings.py for CallbackFilter
and/or require_debug_false to see if that might be the problem.

Cheers,
-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] No index provider 'lucene' found

2011-06-09 Thread Tobias Ivarsson
)
at
 org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:362)
at
 org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
 org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at
 org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:729)
at
 org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
at
 org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at
 org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49)
at
 org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at
 org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
at
 org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:843)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:647)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
at
 org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
at
 org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Generic expansion?

2011-05-20 Thread Tobias Ivarsson
org.neo4j.kernel.Traversal.expanderForAllTypes();

On Fri, May 20, 2011 at 4:53 PM, Marko Rodriguez okramma...@gmail.comwrote:

 Hey,

 Yea. Thats what I'm looking for, but it doesn't seem to be part of the API?
 .. ?

 I can only find StandardExpander (abstract) and OrderedByTypeExpander.

 Thoughts?, -- IM me if you wish.
 Marko.

 On May 20, 2011, at 8:46 AM, Peter Neubauer wrote:

  Marko,
  you could do an AllExpander(Direction.BOTH) or really any subclass of
  Expander, http://api.neo4j.org/current/org/neo4j/graphdb/Expander.html
 
  Does that help?
 
  Cheers,
 
  /peter neubauer
 
  GTalk:  neubauer.peter
  Skype   peter.neubauer
  Phone   +46 704 106975
  LinkedIn   http://www.linkedin.com/in/neubauer
  Twitter  http://twitter.com/peterneubauer
 
  http://www.neo4j.org   - Your high performance graph
 database.
  http://startupbootcamp.org/- Öresund - Innovation happens HERE.
  http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.
 
 
  On Fri, May 20, 2011 at 4:43 PM, Marko Rodriguez okramma...@gmail.com
 wrote:
 
  Hi,
 
  How do I do something like this:
 
PathFinder finder = GraphAlgoFactory.shortestPath(new
  OrderedByTypeExpander().add(DynamicRelationshipType.withName(knows)),
 6);
 
  ...but where I don't care about the RelationshipType taken (and I don't
  want to enumerate all the types I have as I have 100's of types).
 
  Above, when I don't have knows, it doesn't find any paths.
 
  Thanks,
  Marko.
 
  http://markorodriguez.com
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] about relationship whit himself.

2011-05-18 Thread Tobias Ivarsson
Neo4j does not support relationships with the same start node as end node
yet.

This is being worked on however. We actually had it working, passing even
the meanest test cases we could come up with, for a few minutes last friday.
But then we were given a choice between the functionality of these self
relationships, or a substantial performance improvement. We went for the
latter, putting self relationships on hold for a while. We have some ideas
that could make it possible to put self relationships back in, but I don't
know when we can get there.

Cheers,
Tobias

On Wed, May 18, 2011 at 8:54 PM, Jose Angel Inda Herrera 
jai...@estudiantes.uci.cu wrote:

 hello list,
 I have a question you would like to clarify,
 in a graph may have a sibling with himself, that I have it clear.
 neo4j now allow this??, because when I create a relation from ni to nf,
 where ni = nf gives me a exetion,

 thanks very much,
 cheers.
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] GraphRepository parameter

2011-05-17 Thread Tobias Ivarsson
Hi,

Yes, it has to be a special type. Your class has to extend GraphBacked (as
clearly stated by the error you are getting).

The AspectJ compiler will make your class extend GraphBacked automatically
if you have annotated it with either NodeEntity or RelationshipEntity.

The error you are experiencing will usually occur when AspectJ is not
present, are you sure you have set up AspectJ properly in your build
pipeline.

Cheers,
Tobias

On Tue, May 17, 2011 at 2:48 PM, mujer_esponja dracol...@hotmail.comwrote:

 Hello!!
 I have asked another question just a few hours ago, but I am new on neo4j
 and I need some help.

 Now the problem is with this class

 http://static.springsource.org/spring-data/data-graph/docs/current/api/org/springframework/data/graph/neo4j/repository/GraphRepository.html

 In a declaration I need to do: GraphRepositoryMyClass
 as I have seen in examples.

 But I am getting the error:

 Bound mismatch: The type MyClass is not a valid substitute for the bounded
 parameter T extends GraphBacked? of the type
  GraphRepositoryT

 Must it be a special type? Where is the error?? Thanks in advance

 --
 View this message in context:
 http://neo4j-user-list.438527.n3.nabble.com/GraphRepository-parameter-tp2952354p2952354.html
 Sent from the Neo4J User List mailing list archive at Nabble.com.
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Color suggestions for the Self-Relationship bike shed

2011-05-17 Thread Tobias Ivarsson
Having an API that changes behavior based on configuration is a very bad
idea.
It makes the system very hard to work with.

On Tue, May 17, 2011 at 8:30 PM, Craig Taverner cr...@amanzi.com wrote:

 What about a system config enabling/disabling loops? Then we could have
 option 1, but for people that never loops, they can still get the extra
 loop
 check by setting the system config option.


-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Neo4j and Java 1.5

2011-05-16 Thread Tobias Ivarsson
The service code was written before Java 1.5 support was dropped, so that
will still work.

There are a few other places where we use java 6 APIs, mainly
java.util.Queue and java.util.concurrent.TimeUnit. javax.scripting is super
simple to support under 1.5 (there is a backport package), and it is only
used in the server.

Java 1.5 is not supported by Neo4j, since it has reached its End Of Life.

If really need it, you could create your own (unsupported) branch of Neo4j
and modify that yourself to make things work, or pay Neo Technology for
commercial Java 1.5 +Neo4j support.

Cheers,
Tobias

On Sat, May 14, 2011 at 1:04 AM, Peter Neubauer 
peter.neuba...@neotechnology.com wrote:

 Julien,
 I think there are some parts in the server that require Java 1.6 liek the
 javax.scripting , but I am not sure about the kernel. I suspect some of the
 service loader extensions being Java 1.6.

 Anyone else having that info?

 Cheers,

 /peter neubauer

 GTalk:  neubauer.peter
 Skype   peter.neubauer
 Phone   +46 704 106975
 LinkedIn   http://www.linkedin.com/in/neubauer
 Twitter  http://twitter.com/peterneubauer

 http://www.neo4j.org   - Your high performance graph database.
 http://startupbootcamp.org/- Öresund - Innovation happens HERE.
 http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.


 On Fri, May 13, 2011 at 10:40 AM, Julien Serdaru jserd...@gmail.com
 wrote:

  Hi all,
 
 
 
  I am considering using Neo4j embedded into a web application. We have to
  support Java 1.5 since we deploy on client J2EE application servers which
  still run on 1.5.
 
  Looking at the documentation, it seems that Java 1.6 is a prerequisite.
 
 
 
  http://docs.neo4j.org/chunked/stable/deployment-requirements.html
 
 
 
  Are there any plans to maintain 1.5 compatibility?
 
 
 
  Thanks,
 
 
 
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


[Neo4j] Color suggestions for the Self-Relationship bike shed

2011-05-16 Thread Tobias Ivarsson
There has been a few discussions around supporting Loops, relationships with
the same start node as end node, recently. Both here on the mailing list,
and even more at the Neo Technology office.

We have a working patch for handling loops in Neo4j, but one final piece is
missing: what should the API look like for creating a loop? There are three
suggestions for this, I'll list them with the pros and cons we've found for
each. It would be great if you could provide some feedback on what you think
on the matter, and which alternative you prefer.

The alternatives:

1. let the existing createRelationshipTo(Node,RelationshipType) create loops
if the same node is passed as argument.
2. add a new createLoop(RelationshipType) method for creating loops.
3. add a createRelationshipOrLoop(Node,RelationshipType) method that would
work like createRelationshipTo, except it would permit creating loops.


The pros and cons:

PRO 1: does not add a new method to the API that developers have to learn.

CON 1: changes the semantics of the createRelationshipTo method slightly
from what it is today.

CON 1: will not help you catch programming errors where you've mistakenly
written code that creates a relationship to the same node (most of the cases
where code creates relationships to the wrong node it is to the same node).

PRO 2: will let you be explicit of when creating a loop.

PRO 2: will let createRelationshipTo preserve the semantics it has today.
Which will help catching many create relationships to the wrong node cases.

CON 2: will force you to be explicit about loops, most applications that
wan't loops will just treat them as any relationship where the start node
and end node will just happen to be the same.

PRO 3: adds loops as a generic construct (start and end just happens to be
the same) without changing the current semantics of createRelationshipTo

CON 3: Introduces a new method that creates relationships between any two
nodes.


It would of course be possible to go with both 2 and 3, and I think option 3
makes more sense as an addition to option 2, rather than as an alternative
to it.

What do you think?
-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Color suggestions for the Self-Relationship bike shed

2011-05-16 Thread Tobias Ivarsson
Does anyone NOT planning to use loops have an opinion in the matter?
That would be very valuable input.

Cheers,
-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Traversing all relationship types

2011-05-11 Thread Tobias Ivarsson
There is a tentative new traversal API included in Neo4j that features this
functionality:

org.neo4j.kernel.Traversal.description().expand(
org.neo4j.kernel.Traversal.expanderForAllTypes( Direction.OUTGOING ) );

You can read more about it here:
http://wiki.neo4j.org/content/Traversal_Framework

Cheers,
Tobias

On Wed, May 11, 2011 at 9:08 AM, Jean-Pierre Bergamin
jpberga...@gmail.comwrote:

 Hello neo4j users

 I'm just diving into neo4j and playing around with graph algorithms and
 traversers.
 As a start, I just wanted to traverse the whole graph with all relationship
 types in the OUTGOING direction. The traverse() method always expects a
 RelationshipType.
 Is there a simpler way to traverse all RelationshipTypes then building up
 an
 array that contains all types and the outgoing direction (as described here
 http://sujitpal.blogspot.com/2009/06/custom-traverser-for-neo4j.html)?


 Thanks for the help,
 James
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Relationship iterator doesn't implement remove()

2011-05-09 Thread Tobias Ivarsson
As you know it is perfectly valid for an iterator to not implement the
remove() method, the fact of that being a poorly designed API is a separate
issue.

The main reasons for the remove() method not being implemented are:
1. It is very rarely used. Invoking the delete() method on the relationship
object is easy enough anyhow.
2. It adds complexity. In order to support remove() you need to keep track
of the previously returned item to know what to remove, this adds more state
to the iterator and makes the code a bit more complex, which of course means
a higher likelihood of bugs. The cost of this seems a bit high for
supporting a method that is hardly ever used.

Those are the reasons. If you have arguments that counters or invalidates
those reasons please put them forward.

Cheers,
Tobias

On Thu, May 5, 2011 at 5:12 PM, Balazs E. Pataki pat...@dsd.sztaki.huwrote:

 Hi,

 Anothor bug like feature I just found. :-)

 Node#getRelationships() return an Iterablle, from which I can get an
 Iterator. This Iterator (actually IntArrayIterator) however doesn't
 implement the full Iterator interface as it throws an
 UnsupportedOperationException() when remove() is called on it. (APi
 documentationd doesn't mention this restriction)

 Is there any reason why this Iterator cannot implement remove()?

 I now have a wrapper around it, where I call delete() on the last
 Relationship returned by next().

 Regards,
 ---
 balazs
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Retrieving a node linked to the reference node takes forever

2011-05-06 Thread Tobias Ivarsson
Could you provide a stack trace of what the stuck thread is doing?

If you issue 'kill -3' to the process it will print a threaddump to stdout,
that will contain a stack trace for the stuck thread.
If you send that to me it would greatly help me diagnose the issue.

Cheers,
Tobias

On Fri, May 6, 2011 at 12:53 PM, Pablo Pareja ppar...@era7.com wrote:

 Hi !

 I just run into a problem I don't understand at all.
 The scenario is actually quite simple, basically it consists of a simple
 node retrieving through a relationship:

 1. - There is a node (Protein self-interactions node) which is linked to
 the
 reference node by means of one
  relationship (Protein self-interactions relationship). This is the
 piece of code for the node creation/linkage

 public static ProteinSelfInteractionsRel proteinSelfInteractionsRel =
 new ProteinSelfInteractionsRel(null);
...


 proteinSelfInteractionsProperties.put(ProteinSelfInteractionsNode.NODE_TYPE_PROPERTY,
 ProteinSelfInteractionsNode.NODE_TYPE);
long proteinSelfInteractionsNodeId =
 inserter.createNode(proteinSelfInteractionsProperties);
inserter.createRelationship(inserter.getReferenceNode(),
 proteinSelfInteractionsNodeId,
proteinSelfInteractionsRel, null);

 You can check the real code for this file
 here
 https://github.com/pablopareja/Bio4j/blob/master/src/com/era7/bioinfo/bio4j/programs/InitBio4jDB.java
 .
 (line 157)

 2.-  Later on when I try to get the id of that node:

  IterableRelationship iterable =
 manager.getReferenceNode().getRelationships(new
 ProteinSelfInteractionsRel(null),Direction.OUTGOING);
  logger.log(Level.INFO,getting node);
  long proteinSelfInteractionsNodeId =
 iterable.iterator().next().getEndNode().getId();
  logger.log(Level.INFO,done!);

  (You can also check the code for this here
 
 https://github.com/pablopareja/Bio4j/blob/master/src/com/era7/bioinfo/bio4j/programs/ImportProteinInteractions.java
 (line
 117)

  It just gets stuck forever after getting node

 There's no exception thrown and the program just seems to freeze forever
 (I've already been waiting for more than 1h with no success)

 Do you have any ideas of what could be going on??

 I really have no clue of what could I be doing wrong...
 In case it may be useful, the size of the DB at this point is around
 46.000.000 nodes and 380.000.000 relationships (only a dozen or so
 coming/going to
 the reference node)

 Thanks in advance,


 Pablo

 --
 Pablo Pareja Tobes
 LinkedInhttp://www.linkedin.com/in/pabloparejatobes
 Twitter   http://www.twitter.com/pablopareja

 http://about.me/pablopareja
 http://www.ohnosequences.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Regarding Weighted Node

2011-05-03 Thread Tobias Ivarsson
Hi Pooja,

You could add the weight as a property on the nodes.

For inserting the nodes in a particular order, if you have a complete
dataset when you insert you can always sort it before you insert it. Or did
I misunderstand your question?

Cheers,
Tobias

On Tue, May 3, 2011 at 7:24 PM, pooja naik npooj...@yahoo.com wrote:

 Hi

 Is there a way in neo 4j to add weights to the node ?

 I want to add weights to the node and want to insert and delete nodes in
 the increasing order of weight?

 For example:

 The following is the graph/tree representation I am interested in.

 Parent
 node 1 - weight 1

 Has children with weights

 Node 2 - weight 3, node 3- weight 5






 So the insertion of node4 -weight 4 will go before node3- weight 5 and
 after node 2- 3



 Regards
 Pooja
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] neo4j.py + django relationship to same node type

2011-04-29 Thread Tobias Ivarsson
Pythons evaluation model prevents you from referencing things that have not
been defined yet.
In this case you are trying to reference Person before the class definition
of Person is done.
Just like in regular Django models this can be worked around by just
providing the name of the class as a string. Then the actual class will be
looked up at a later time.

Cheers,
tobias

On Thu, Apr 28, 2011 at 10:31 PM, Hammad hammad.ul...@gmail.com wrote:

 hey it my first try at using django + neo4j together. and i have already
 run
 into a problem:

 here is my model:

 from neo4j.model import django_model as neoModel

 class Person(neoModel.NodeModel):
name = neoModel.Property(indexed=True)
age = neoModel.Property()
related = neoModel.Relationship(Person, type=neoModel.Outgoing.related)
href = property(lambda self: ('/idea/%s/' % (self.node.id,)))

def __unicode__(self):
return u'%s' % (self.name)

 while testing it by starting the shell from manage.py i get this error when
 i import the models(from myapp import models):

 Traceback (most recent call last):
  File console, line 1, in module
  File /home/hammad/Dev/Project/Myproj/myapp/models.py, line 4, in
 module
class Person(neoModel.NodeModel):
  File /home/hammad/Dev/Project/Myproj/myapp/models.py, line 7, in Person
related = neoModel.Relationship(Person, type=neoModel.Outgoing.related)
 NameError: name 'Person' is not defined

 any ideas on how i can create a relationship between two nodes of the same
 time. Its such a basic use-case that i am sure i am missing something...

 --
 Hammad Ullah
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Neo Memory Tuning - Autotuning

2011-04-29 Thread Tobias Ivarsson
This is roughly what the built in auto tuning does.
In what ways is that insufficient for your needs?

Cheers,
Tobias

On Fri, Apr 29, 2011 at 3:43 PM, Rick Bullotta
rick.bullo...@thingworx.comwrote:

 Is there any suggested way to auto-tune the various Neo memory mapping
 settings?  We have such a wide variety of database scenarios and system
 configurations, that it is not possible for us to have a static set of
 configuration parameters, but we'd like to avoid requiring human
 intervention.

 Curious as to if others have faced this challenge and how they solved it.
  I was thinking about asking the JVM how much memory was allocated (using
 the Runtime class) and checking the file sizes of the various Neo stores to
 prepare a HashMap with the appropriate mapped memory settings prior to
 starting the EmbeddedGraphDatabase.  I was going to allocate memory mapping
 based on proportional sizes of the node and relationship db files.

 I have to think this would be generally useful capability, so I'm wondering
 if anyone has already implemented something along those lines.

 Thanks,

 Rick

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Neo Memory Tuning - Autotuning

2011-04-29 Thread Tobias Ivarsson
Auto tuning is what happens when you don't specify any memory mapping
parameters.
Default values was a long time ago.

-tobias

On Fri, Apr 29, 2011 at 5:41 PM, Rick Bullotta
rick.bullo...@thingworx.comwrote:

 Cool! Didn't even know it was in there! I thought it used the default
 values unless told otherwise.  How can auto tuning be enabled?


 -Original Message-
 From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org]
 On Behalf Of Tobias Ivarsson
 Sent: Friday, April 29, 2011 11:40 AM
 To: Neo4j user discussions
 Subject: Re: [Neo4j] Neo Memory Tuning - Autotuning

 This is roughly what the built in auto tuning does.
 In what ways is that insufficient for your needs?

 Cheers,
 Tobias

 On Fri, Apr 29, 2011 at 3:43 PM, Rick Bullotta
 rick.bullo...@thingworx.comwrote:

  Is there any suggested way to auto-tune the various Neo memory mapping
  settings?  We have such a wide variety of database scenarios and system
  configurations, that it is not possible for us to have a static set of
  configuration parameters, but we'd like to avoid requiring human
  intervention.
 
  Curious as to if others have faced this challenge and how they solved it.
   I was thinking about asking the JVM how much memory was allocated (using
  the Runtime class) and checking the file sizes of the various Neo stores
 to
  prepare a HashMap with the appropriate mapped memory settings prior to
  starting the EmbeddedGraphDatabase.  I was going to allocate memory
 mapping
  based on proportional sizes of the node and relationship db files.
 
  I have to think this would be generally useful capability, so I'm
 wondering
  if anyone has already implemented something along those lines.
 
  Thanks,
 
  Rick
 
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 



 --
 Tobias Ivarsson tobias.ivars...@neotechnology.com
 Hacker, Neo Technology
 www.neotechnology.com
 Cellphone: +46 706 534857
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Neo Memory Tuning - Autotuning

2011-04-29 Thread Tobias Ivarsson
Right. Thanks for pointing out that there is a faulty documentation in the
wiki. I'll add it to the backlog of documentation fixes.

Have a great weekend.
-tobias

On Fri, Apr 29, 2011 at 5:52 PM, Rick Bullotta
rick.bullo...@thingworx.comwrote:

 This was a case of RTFM, or rather RTWM (read the wrong manual) - the old
 Neo4J Wiki stuff on Configuration Settings and Performance still comes up
 first on search results, thus my confused questions.


 -Original Message-
 From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org]
 On Behalf Of Tobias Ivarsson
 Sent: Friday, April 29, 2011 11:46 AM
 To: Neo4j user discussions
 Subject: Re: [Neo4j] Neo Memory Tuning - Autotuning

 Auto tuning is what happens when you don't specify any memory mapping
 parameters.
 Default values was a long time ago.

 -tobias

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Neo4j on OSGi

2011-04-28 Thread Tobias Ivarsson
Right,

OSGiExtensionLoader is not completed, but it is supposed to work roughly as
you have outlined.

-tobias

On Wed, Apr 27, 2011 at 1:07 PM, Nicolas Jouanin
nicolas.joua...@gmail.comwrote:

 Hi Andreas,

 I'm currently looking at class OSGiExtensionLoader and I wonder what's the
 final purpose of this class ?
 I mean currently it's used to load extensions which are registered as OSGi
 services through the loadExtensionsOfType method.
 Also it manages a list of bundle but does nothing with it. I think there's
 something missing here. I guess it should introspect theses bundle and
 register services declared in META-INF/services/ . Is that it ?

 Nicolas.

 2011/4/25 Andreas Kollegger andreas.kolleg...@neotechnology.com

  Hi Nicolas,
 
  I had done some work with embedding an OSGi environment within the server
  to host plugins, hoping to slowly modularize the server in a sensible
 way.
  But that code has been pulled out, as we opted for the Java ServiceLoader
  approach for plugins.
 
  When deploying into an OSGi environment, I've approached it as embedded
  development. The server only adds the REST api (which isn't needed when
  talking directly to the database), and the Webadmin (replaced by the OSGi
  webconsole. well, except for visualization). We've been relaxed about
 good
  OSGi discipline in the server bits, because it seemed like OSGi
 development
  is embedded development.
 
  Are you developing an application in OSGi, or are you interested in
  replicating the server (including REST+webadmin) in OSGi?
 
  -Andreas
 
  On Apr 25, 2011, at 5:50 AM, Nicolas Jouanin wrote:
 
   Hi,
  
   I think an OSGi section in the Neo4j wiki would be great.
   Of course there's no problem for adding content from my documentation
  into Neo4j wiki. Let me know if you want me to do it.
   Concerning OSGi support for the server, I'm also working on it.
 Currently
  I'm trying to make the server component run using HttpService for
  deployment, and use OSGi services to discover the graphDb and server
  modules.
   I have some troubles because of server and server-api packages which
  overlap and protected or even private method or class declaration.
   You can follow the work on github :
  https://github.com/njouanin/community  (currently working on move-server
  branch).
  
   Nicolas.
  
   Le 24 avr. 2011 à 23:17, Peter Neubauer a écrit :
  
   Nicolas,
   Great and concise write-up! Would it be ok to include something along
   these lines into the documentation for neo4j that gets compiled into
   http://docs.neo4j.org/ ? An OSGi related section is certainly a good
   idea.
  
   Andreas Kollegger has been playing with the OSGi side of things
   regarding the server, but I don't know the state of it at the moment.
   Andreas, any status?
  
   Cheers,
  
   /peter neubauer
  
   GTalk:  neubauer.peter
   Skype   peter.neubauer
   Phone   +46 704 106975
   LinkedIn   http://www.linkedin.com/in/neubauer
   Twitter  http://twitter.com/peterneubauer
  
   http://www.neo4j.org   - Your high performance graph
  database.
   http://startupbootcamp.org/- Öresund - Innovation happens HERE.
   http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing
 party.
  
  
  
   On Mon, Apr 18, 2011 at 2:40 PM, Nicolas Jouanin
   nicolas.joua...@gmail.com wrote:
   Hi,
  
   I've written a step-by-step guide for installing Neo4j on OSGi (
  
  http://wiki.escapek.org/display/DEV/Neo4j+community+edition+installation
 ).
   It is based on my own experience and currently it shows how to
 install
  Neo4j
   community edition and opening a Embedded Graph Database from another
  bundle.
   A small example is provided.
   Next step i'd like to achieve would be a be able to only to install
  server,
   webadmin or ha features but also make them running using OSGi
  HTTPService
   for example. Those features could also for example look for exposed
   GraphDatabase and then register management of webadmin features
   automatically.
   I don't know if it would be useful or even if that makes sense
  regarding
   neo4j use cases. Let me know what you think about that.
  
   Nicolas.
   ___
   Neo4j mailing list
   User@lists.neo4j.org
   https://lists.neo4j.org/mailman/listinfo/user
  
   ___
   Neo4j mailing list
   User@lists.neo4j.org
   https://lists.neo4j.org/mailman/listinfo/user
  
   ___
   Neo4j mailing list
   User@lists.neo4j.org
   https://lists.neo4j.org/mailman/listinfo/user
 
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology

Re: [Neo4j] ClosedChannelExceptions in highly concurrent environment

2011-04-28 Thread Tobias Ivarsson
Hi Jennifer,

I'd first like to thank you for the testing and analysis you've done. Very
useful stuff. Do you think you could send some test code our way that
reproduces this issue?

This is actually the first time this issue has been reported, so I wouldn't
say it is a common issue. My guess is that your thread volume triggered a
rare condition that wouldn't be encountered otherwise.

I'm also curious to know how you found the source of the interruptions. When
I debug thread interruptions I've never been able to find out where the
thread got interrupted from without doing tedious procedures of breakpoint +
logging + trying to match thread ids. If you have a better method for doing
that I'd very much like to know.

I think we should focus the effort on fixing the interruption issue if we
can. And I believe we would be able to do that if the interruptions do in
fact originate from where you say they do. But the suggestion of being able
to switch the lucene directory implementation is still interesting, but as
you point out since it has issues on some platforms it would be better if we
could be rid of the interruption issue.

Cheers,
Tobias

On Thu, Apr 28, 2011 at 12:41 AM, Jennifer Hickey jhic...@vmware.comwrote:

 Hello,
 I've been running some tests w/approx 400 threads reading various indexed
 property values.  I'm running on 64 bit Linux.  I was frequently seeing the
 ClosedChannelException below.  The javadoc on Lucene's NIOFSDirectory states
 that Accessing this class either directly or indirectly from a thread while
 it's interrupted can close the underlying file descriptor immediately if at
 the same time the thread is blocked on IO. The file descriptor will remain
 closed and subsequent access to {@link NIOFSDirectory} will throw a {@link
 ClosedChannelException}.  If your application uses either {@link
 Thread#interrupt()} or {@link Future#cancel(boolean)} you should use {@link
 SimpleFSDirectory} in favor of {@link NIOFSDirectory}.

 A bit of debugging revealed that the Thread.interrupts were coming from
 Neo4j, specifically in RWLock and MappedPersistenceWindow.  So it seems like
 this would be a common problem, though perhaps I am missing something?

 SimpleFSDirectory seems a bit of a performance bottleneck, so I switched to
 MMapDirectory and the problem did go away.  I didn't see a way to switch
 implementations w/out modifying neo4j code, so I changed LuceneDataSource as
 follows:

  static Directory getDirectory( String storeDir,
IndexIdentifier identifier ) throws IOException
 {
MMapDirectory dir=new MMapDirectory(getFileDirectory( storeDir,
 identifier), null);
if(MMapDirectory.UNMAP_SUPPORTED) {
dir.setUseUnmap(true);
}
return dir;
  }

 So I'm wondering if others have seen this problem and/or if there is a
 recommended solution?  Our product runs on quite a few different operating
 systems, so I have some reservations about using MMapDirectory as well
 (javadoc speaks of a few caveats on Windows, 64 vs 32, etc). Also, I'd
 rather not maintain a patched version of the neo4j code if avoidable.

 Thanks!
 Jennifer

 Exception:
 Caused by: java.nio.channels.ClosedChannelException
 at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88)
 at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:613)
 at
 org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:161)
 at
 org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:139)
 at
 org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:285)
 at
 org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:160)
 at
 org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39)
 at org.apache.lucene.store.DataInput.readVInt(DataInput.java:86)
 at
 org.apache.lucene.index.codecs.DeltaBytesReader.read(DeltaBytesReader.java:40)
 at
 org.apache.lucene.index.codecs.PrefixCodedTermsReader$FieldReader$SegmentTermsEnum.next(PrefixCodedTermsReader.java:469)
 at
 org.apache.lucene.index.codecs.PrefixCodedTermsReader$FieldReader$SegmentTermsEnum.seek(PrefixCodedTermsReader.java:385)
 at org.apache.lucene.index.TermsEnum.seek(TermsEnum.java:68)
 at org.apache.lucene.index.Terms.docFreq(Terms.java:53)
 at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:898)
 at org.apache.lucene.index.IndexReader.docFreq(IndexReader.java:882)
 at
 org.apache.lucene.index.DirectoryReader.docFreq(DirectoryReader.java:687)


 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] REST results pagination

2011-04-22 Thread Tobias Ivarsson
On Thu, Apr 21, 2011 at 11:18 PM, Michael Hunger 
michael.hun...@neotechnology.com wrote:

 Rick,

 great thoughts.

 Good catch, forgot to add the in-graph representation of the results to my
 mail, thanks for adding that part. Temporary (transient) nodes and
 relationships would really rock here, with the advantage that with HA you
 have them distributed to all cluster nodes.
 Certainly Craig has to add some interesting things to this, as those
 resemble probably his in graph indexes / R-Trees.

 As traversers are lazy a count operation is not so easily possible, you
 could run the traversal and discard the results. But then the client could
 also just pull those results until it reaches its
 internal tresholds and then decide to use more filtering or stop the
 pulling and ask the user for more filtering (you can always retrieve n+1 and
 show the user that there are more that n results available).

 The index result size() method only returns an estimate of the result size
 (which might not contain currently changed index entries).

 Please don't forget that a count() query in a RDBMS can be as ridicully
 expensive as the original query (especially if just the column selection was
 replaced with count, and sorting, grouping etc was still left in place
 together with lots of joins).

 Sorting on your own instead of letting the db do that mostly harms the
 performance as it requires you to build up all the data in memory, sort it
 and then use it. Instead of having the db do that more efficiently, stream
 the data and you can use it directly from the stream.


throw new SlapOnTheFingersException(sometimes the application developer can
do a better job since she has better knowledge of the data, the database
only has generic knowledge);

Since Jake had already mentioned (in this very thread) that he expected one
of those, I thought I might as well throw one in there.

I agree with the analysis of count(), as the name (count) implies, it will
have to run the entire query in order to count the number of resulting
items.

About sorting I'm torn. The perception of sorting in the database being slow
that Rick points to is one that I've seen a lot. When you hand the
responsibility of sorting to the database you hide the fact that sorting is
an expensive operation, it does require reading in all data in order to sort
it. People often expect databases to be smarter than that, since they
sometimes are, but that is pretty much only when reading straight from an
index and not doing much more. A generic sort of data can never be better
than O(log(n!)) [O(log(n!)) is almost equal to, and commonly rounded to the
easier to compute function O(n log(n))]. If you put the responsibility of
sorting in the hands of the application you can sometimes utilize knowledge
about the data to do a more efficient sorting than the database could have
done. Most often by simply doing an application level filtering of the data
before sorting it, based on some filtering that could not be transfered to
the database query. This does however make the work of the application
developer slightly more tedious, which is why I think it would be sensible
to have support for sorting on the database level, and hope that users will
be sensible about using it, and not assume magic from it.

Something I find very interesting is the concept of semi-sorted data.
Semi-sorted data is often good enough, easier to achieve, and quite easy to
then sort completely if that is required. Examples of semi-sorted data could
be data in an order that satisfies the heap property. Or for spatial queries
returning the closest hits first, but not necessarily in perfect order, say
returning the hits within a miles radius first, before the ones in a radius
between 1-10 miles, and so on, without requiring the hits in each 'segment'
to be perfectly ordered by distance. Breadth first order is another example
of semi-sorted data, that could be used when traversing data as you've
outlined with paging nodes, or similarly grouped by parent node-order.

I must say that I really enjoy following this discussion. I really like the
idea of streaming, since I think that can be implemented more easily than
paging, while satisfying many of the desired use cases. But I still want to
hear more arguments for and against both alternatives. And as has already
been pointed out, they aren't mutually exclusive.

I'll keep listening in on the conversation, but I don't have much more to
add at this point. I have one desire for the structure of the conversation
though. When you quote what someone else has said before you, could you
please include who that person was, it makes going back and reading the full
context easier.

Cheers,
-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Multitenancy in neo4j

2011-04-18 Thread Tobias Ivarsson
If you by multitenancy simply mean that you want to have multiple separate
graph database, you can do that as easily as:
GraphDatbaseService graphdb1 = new EmbeddedGraphDatabase(
/path/to/your/first/graphdb );
GraphDatbaseService graphdb2 = new EmbeddedGraphDatabase(
/path/to/your/second/graphdb );

Since that is already possible in the embedded scenario, the multitencncy
support that has been discussed for the future is a server specific feature.

That work for the server might bring some new capabilities to the embedded
setup as well, but the main part of it is still going to be
as exemplified above.

Or you could of course do as Jim suggested and make the sub-graph
distinction an application level separation.

One thing to keep in mind when starting multiple EmbeddedGraphDatabase
instances in the same JVM (and on the same computer for that matter) is that
the automatic configuration will assume that it is the only graph database
running on that computer, sou you will need to supply manual configuration.
See: http://wiki.neo4j.org/content/Configuration_Settings

Cheers,
Tobias

On Mon, Apr 18, 2011 at 10:21 AM, ntankovic nikola.tanko...@gmail.comwrote:

 Hi all,

 I'm facing a situation to develop a multitenant graph environment. I heard
 from some of you guys that this is planned for neo4j server. Is this neo4j
 server only or will I also be possible to use this on neo4j embedded
 engine?

 --
 View this message in context:
 http://neo4j-user-list.438527.n3.nabble.com/Multitenancy-in-neo4j-tp2833480p2833480.html
 Sent from the Neo4J User List mailing list archive at Nabble.com.
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Strange performance difference on different machines

2011-04-18 Thread Tobias Ivarsson
Hi Bob,

What happens here is that you perform a tiny operation in each transaction,
so what you are really testing here is how fast your file system can flush,
because with such tiny transactions all of the time is going to be spent in
transactional overhead (i.e. flushing transaction logs to the disk).

The reason you see such large differences between Mac OS X and Linux is
because Mac OS X cheats. Flushing a file (fdatasync) on Mac does pretty much
nothing. The only thing Mac OS X guarantees is that it will write the data
that you just flushed before it writes the next data block you flush, so
called ordered writes. This means that you could potentially get data-loss
on hard failure, but never in a way that makes your data internally
inconsistent.

So to give a short answer to your questions:
1) The linux number is reasonable, Mac OS X cheats.
2) What you are testing is the write speed of your disk for writing small
chunks of data.

Cheers,
Tobias

On Mon, Apr 18, 2011 at 10:57 PM, Bob Hutchison hutch-li...@recursive.cawrote:

 Hi,

 Using Neo4j 1.3 and the Borneo (Clojure) wrapper I'm getting radically
 different performance numbers with identical test code.

 The test is a simple-minded: create two nodes and a relation between them.
 No properties, no indexes, all nodes and relations are different.

 On OS X, it takes about 50s to perform that operation 50,000 times,  0.8s
 to do it 500 times. It uses roughly 30-40% of one core to do this.

 On linux it takes about 30s to perform that operation 500 times. The CPU
 usage is negligible (really negligible... almost none).

 I cannot explain the difference in behaviour.

 I have two questions:

 1) is either of these a reasonable number? I hoping the OS X numbers are
 not too fast.

 2) any ideas as to what might be the cause of this?

 The Computers are comparable. The OS X is a 2.8 GHz i7, the linux box is a
 3.something GHz Xeon (I don't remember the details).

 Thanks in advance for any help,
 Bob

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Architecturing Neo4J database

2011-04-17 Thread Tobias Ivarsson
What is the required number of relationships?

I'll answer your questions inline below.

On Sun, Apr 17, 2011 at 7:27 PM, Dima Gutzeit
dima.gutz...@mailvision.comwrote:

 Oops, small correction. The required number of nodes is 2 billions and not
 100 millions as originally stated.

 On Sun, Apr 17, 2011 at 8:16 PM, Dima Gutzeit
 dima.gutz...@mailvision.comwrote:

  Dear list members,
 
  I have a need for storage engine (graph based) for a server I am
 building.
 
  Here are the requirements:
 
  1. Support for High Availability


You can tick that box with Neo4j :)


   2. Ability to store around 100 millions of nodes, estimated data size of
  around 5 terabytes.


Measuring data in bytes is blunt and difficult. Statistically, all users
trying to do that before using Neo4j for at least a while has gotten it
wrong in one way or another. Is it ok if I take this figure with a grain of
salt? Your statement of 2 bilion nodes is more helpful, add to that the
estimated number of relationships (either per node or in total) and the
estimated number of properties for each node and relationship and we'll have
a better model for estimating both on-disk size and how to best configure
your Neo4j installation for optimal cache utilization.


 
  I've been reading about different options that Neo4J provides and I need
 to
  make a final decision about the layout, few questions:
 
  - I expect mostly-read-access to the database and I've been reading about
  the HA approach, by replicating the data across several nodes. Is there
 an
  option to use a centralized storage server (NAS/SAN) and working with
  several nodes (for scaling) but without data replication ?


No. Since Neo4j requires exclusive file locks for the storage files to
guarantee write consistency.

It also doesn't work well even if you have a single write-instance and let
all other instances be read-only instances used for only scaling reads. This
is because Neo4j relies heavily on caching to get good read performance. If
the underlying storage data is changed by the write-master the read slaves
will not know about it and thus suffer from stale caches which can lead to
an inconsistent view of the graph.

The Neo4j HA solution provides the appropriate replication and cache
invalidation to make this work consistently across multiple instances.


  
  - The documentation describes HA installation with embedded neo4j
 instance,
  can the same be configured for a standalone server ?


Yes.


  
  - If using an embedded version of the database, can still the webadmin be
  used ?


Not yet. This is being considered for future versions.


  
  - What is supported request rate / throughput of a single Neo4j server,
 is
  there benchmarking data I can consult with including the hardware config
 of
  a testbed ?


The figures for an embedded Neo4j setup is between 1000 to 2000
relationships traversed per millisecond for fully cached data.
With approximately 10% of that for data only in the low level (file system)
cache. The server setup used the same graph database internally, but adds
some per-request network overhead to that. I don't have any benchmark
figures. Could someone fill in?

Cheers,
-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Mac Pro Question ...

2011-04-15 Thread Tobias Ivarsson
There are known issues with neo4j install on the mac. We are working on
that.

But just running should be fine, i.e. neo4j start, I do that on my mac all
the time. Roughly half of the Neo Technology devteam have Macs as their main
development machines.

Cheers,
Tobias

On Fri, Apr 15, 2011 at 6:19 AM, Rense Lange rensela...@earthlink.netwrote:




 Is there anyone who is running neo4j on their Mac? How did you do it? I
 have a Mac pro, and I can't seem to get it going. Right now, the
 installation methods provided on the web don't work. I hope there is a way
 around this. Help!

 Rense Lange

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] 20 billion primitives machine

2011-04-14 Thread Tobias Ivarsson
Hi Jamie,

Very interesting use case you have there.

If you could just provide a few more bits of information about your data,
I'd be able to provide a better analysis.

* Is the distribution of relationships uniform across the nodes. If not, how
much does it vary?

* What kind of operation you want to do in the 50ms latency that you've
specified. You just mentioned relatively shallow traversals, approximately
how deep is that? The key figure to get at is how many relationships you
need to traverse, and that is a function of the depth of the traversal and
the distribution of the relationships.

* Is the 50ms latency for each single request? What is the estimated number
of concurrent requests?

* What kind of properties are those 32 byte values? ASCII strings of length
32?

* The bulk loading phase you describe, is it a one time import of initial
data, or a regularly recurring thing?

* In the bulk load phase, what form does the data have? Would it be possible
to have that data in a format where each node is uniquely identified by a
number (in both the node data listing, and the edge-list).


Finally, do you have test data for these sizes? If possible I'd love to work
with you on this, to get an good use case to work on for improving the large
data story in Neo4j.

Cheers,
Tobias

On Thu, Apr 14, 2011 at 3:12 PM, Jamie Stephens j...@morphism.com wrote:

 Folks,

 I've got an application that has (will have) about 2 billion vertexes
 and maybe 8 billion edges (?).  Maybe an avg of 4 properties per
 vertex -- with maybe an avg of 32 bytes/value.  So I guess that's 16
 billion primitives.  Let's round to 20 billion.  My edges estimate is
 a relatively uninformed guess.  Just starting to dig into the data.

 Traversals will be relatively shallow.  Concurrent access.  Throughput
 is more important than latency.  But latency should be better than
 maybe 50ms 99% of the time (allowing for some cache warming and some
 GC).  I don't know much yet about locality.  I'm not sure yet how
 sensitive the app will be to long GCs.

 We will need to do a big batch load, and writes will need to be fast
 in that phase.  After that, we'll see more reads that writes.  So I
 imagine a config for the batch load and another config for production.

 I understand cache sharding, application-level partitioning, and so
 forth.  I'm wondering what I can do on a single machine -- and what
 that machine should look like.

 http://docs.neo4j.org/chunked/stable/configuration-jvm.html and
 http://wiki.neo4j.org/content/Neo4j_Performance_Guide are encouraging.
  And having knobs as documented at
 http://wiki.neo4j.org/content/Configuration_Settings is great.  Nice
 work!

 I'm hoping I might be able to get away with 128GB RAM on 12 cores with
 data striped over a handful of disks (SSDs if required).  We'll
 probably also need a cluster for both traffic and availability, but
 that's another topic.

 Does anybody have experience with a data set like this on a similar
 machine?  How much RAM and how much disk -- and what kinds and in what
 configuration?  Latency, throughput, general experience?  Any
 production deployments?

 I'd appreciate any guidance or feedback.  I'm happy to summarize later
 if that'd be helpful.

 BTW, my testbed uses Clojure with clojure.contrib.server-socket and
 https://github.com/wagjo/borneo. Very convenient!

 --Jamie
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Neo Server slow on frequent requests

2011-04-13 Thread Tobias Ivarsson
Hi Dario,

This dump looks perfectly fine, the expected threads are there, but they are
all idle waiting for work.

When I asked for a thread dump, I wanted one from when the server was under
load and you experienced problems. Sorry for not being clear about that.

Cheers,
Tobias

On Wed, Apr 13, 2011 at 10:34 AM, Dario Rexin dario.re...@xing.com wrote:

 Hey,

 Somehow my attached files always get deleted. Heres the dump:

 https://gist.github.com/917199


 Cheers,

 Dario

 Am 13.04.11 10:30 schrieb Dario Rexin unter dario.re...@xing.com:

  Hi Tobias,
 
  Here's the thread dump you asked for. Thank you for taking a look at
 this.
 
 
  Cheers,
 
  Dario
 
 
  Am 12.04.11 22:16 schrieb Tobias Ivarsson unter
  tobias.ivars...@neotechnology.com:
 
  Hi Dario,
 
  Looking at that picture it is indeed clear that a number of threads are
  waiting for something. What is not shown is the more important
 information
  about *what* they are waiting for. I would love to get information like
 that
  in order to investigate the cause of the performance problem you are
 seeing.
  If you could send a thread dump instead of a screenshot that would be a
 lot
  more useful, since that would contain information about contention that
 I
  could actually analyze. The easiest way to get a thread dump is by
 sending
  the SIGQUIT signal (kill -3) to the JVM process running Neo4j.
 
  Cheers,
  Tobias
 
  On Tue, Apr 12, 2011 at 6:35 PM, Dario Rexin dario.re...@xing.com
 wrote:
 
  Hi all,
 
  Due to huge performance issues with some of our neo queries I profiled
 my
  calls on the neo server. The profiling shows, that up to 85% of the
 time the
  threads are waiting for other threads. I don¹t understand what¹s going
 on
  there. Hopefully someone with a deeper knowledge can help me. Am I
 doing
  something wrong, or is it normal, that most of the time the threads are
  blocking each other? Her is a screenshot, showing the results of my
  profiling: http://i.imgur.com/eIfam.jpg
 
 
  Thanks in advice,
 
  Dario
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 
 
 

 --
 Dario Rexin
 Junior Manager Engineering
 dario.re...@xing.com

 XING AG
 Gaensemarkt 43, 20354 Hamburg, Germany

 Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807
 Exec. Board (Vorstand): Dr. Stefan Groß-Selbeck (Vorsitzender), Ingo Chu,
 Dr. Helmut Becker, Jens Pape
 Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Dr. Neil
 Sunderland

 Please join my network on XING:
 https://www.xing.com/profile/Dario_Rexin

 This e-mail may contain confidential and/or privileged information. If you
 are not the intended recipient (or have received this e-mail in error)
 please notify the sender immediately and destroy this e-mail. Any
 unauthorised copying, disclosure or distribution of the material in this
 e-mail is strictly forbidden and may be unlawful.

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Neo Server slow on frequent requests

2011-04-13 Thread Tobias Ivarsson
That one looks much more interesting. I'm too tired right now, but I'll look
at it tomorrow.

On Wed, Apr 13, 2011 at 11:50 AM, Dario Rexin dario.re...@xing.com wrote:

 Hi Tobias,

 I was already sending requests to the server in the last dump. Here is
 another, hopefully this one is more helpful. The longer I request data, the
 longer it takes for the server to answer. After some time it frequently
 freezes for up to several seconds without answering to any of the requests.

 https://gist.github.com/917283

 Cheers,

 Dario

 Am 13.04.11 11:14 schrieb Tobias Ivarsson unter
 tobias.ivars...@neotechnology.com:

  Hi Dario,
 
  This dump looks perfectly fine, the expected threads are there, but they
 are
  all idle waiting for work.
 
  When I asked for a thread dump, I wanted one from when the server was
 under
  load and you experienced problems. Sorry for not being clear about that.
 
  Cheers,
  Tobias
 
  On Wed, Apr 13, 2011 at 10:34 AM, Dario Rexin dario.re...@xing.com
 wrote:
 
  Hey,
 
  Somehow my attached files always get deleted. Heres the dump:
 
  https://gist.github.com/917199
 
 
  Cheers,
 
  Dario
 
  Am 13.04.11 10:30 schrieb Dario Rexin unter dario.re...@xing.com:
 
  Hi Tobias,
 
  Here's the thread dump you asked for. Thank you for taking a look at
  this.
 
 
  Cheers,
 
  Dario
 
 
  Am 12.04.11 22:16 schrieb Tobias Ivarsson unter
  tobias.ivars...@neotechnology.com:
 
  Hi Dario,
 
  Looking at that picture it is indeed clear that a number of threads
 are
  waiting for something. What is not shown is the more important
  information
  about *what* they are waiting for. I would love to get information
 like
  that
  in order to investigate the cause of the performance problem you are
  seeing.
  If you could send a thread dump instead of a screenshot that would be
 a
  lot
  more useful, since that would contain information about contention
 that
  I
  could actually analyze. The easiest way to get a thread dump is by
  sending
  the SIGQUIT signal (kill -3) to the JVM process running Neo4j.
 
  Cheers,
  Tobias
 
  On Tue, Apr 12, 2011 at 6:35 PM, Dario Rexin dario.re...@xing.com
  wrote:
 
  Hi all,
 
  Due to huge performance issues with some of our neo queries I
 profiled
  my
  calls on the neo server. The profiling shows, that up to 85% of the
  time the
  threads are waiting for other threads. I don¹t understand what¹s
 going
  on
  there. Hopefully someone with a deeper knowledge can help me. Am I
  doing
  something wrong, or is it normal, that most of the time the threads
 are
  blocking each other? Her is a screenshot, showing the results of my
  profiling: http://i.imgur.com/eIfam.jpg
 
 
  Thanks in advice,
 
  Dario
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 
 
 
 
  --
  Dario Rexin
  Junior Manager Engineering
  dario.re...@xing.com
 
  XING AG
  Gaensemarkt 43, 20354 Hamburg, Germany
 
  Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807
  Exec. Board (Vorstand): Dr. Stefan Groß-Selbeck (Vorsitzender), Ingo
 Chu,
  Dr. Helmut Becker, Jens Pape
  Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Dr. Neil
  Sunderland
 
  Please join my network on XING:
  https://www.xing.com/profile/Dario_Rexin
 
  This e-mail may contain confidential and/or privileged information. If
 you
  are not the intended recipient (or have received this e-mail in error)
  please notify the sender immediately and destroy this e-mail. Any
  unauthorised copying, disclosure or distribution of the material in this
  e-mail is strictly forbidden and may be unlawful.
 
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 
 
 

 --
 Dario Rexin
 Junior Manager Engineering
 dario.re...@xing.com

 XING AG
 Gaensemarkt 43, 20354 Hamburg, Germany

 Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807
 Exec. Board (Vorstand): Dr. Stefan Groß-Selbeck (Vorsitzender), Ingo Chu,
 Dr. Helmut Becker, Jens Pape
 Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Dr. Neil
 Sunderland

 Please join my network on XING:
 https://www.xing.com/profile/Dario_Rexin

 This e-mail may contain confidential and/or privileged information. If you
 are not the intended recipient (or have received this e-mail in error)
 please notify the sender immediately and destroy this e-mail. Any
 unauthorised copying, disclosure or distribution of the material in this
 e-mail is strictly forbidden and may be unlawful.

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman

Re: [Neo4j] Neo Server slow on frequent requests

2011-04-12 Thread Tobias Ivarsson
Hi Dario,

Looking at that picture it is indeed clear that a number of threads are
waiting for something. What is not shown is the more important information
about *what* they are waiting for. I would love to get information like that
in order to investigate the cause of the performance problem you are seeing.
If you could send a thread dump instead of a screenshot that would be a lot
more useful, since that would contain information about contention that I
could actually analyze. The easiest way to get a thread dump is by sending
the SIGQUIT signal (kill -3) to the JVM process running Neo4j.

Cheers,
Tobias

On Tue, Apr 12, 2011 at 6:35 PM, Dario Rexin dario.re...@xing.com wrote:

 Hi all,

 Due to huge performance issues with some of our neo queries I profiled my
 calls on the neo server. The profiling shows, that up to 85% of the time the
 threads are waiting for other threads. I don’t understand what’s going on
 there. Hopefully someone with a deeper knowledge can help me. Am I doing
 something wrong, or is it normal, that most of the time the threads are
 blocking each other? Her is a screenshot, showing the results of my
 profiling: http://i.imgur.com/eIfam.jpg


 Thanks in advice,

 Dario
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Neo4j Storage provider

2011-04-10 Thread Tobias Ivarsson
Hi Elangovan,

In theory it would be possible. Neo4j started out as an abstraction layer on
top of other storage systems many years ago. The first implementation that
was somewhat useable was based on Berkley DB for storage. But even then
performance wasn't good enough. Not until we implemented our own storage
technology was it good enough for actual use. Since then evolution of Neo4j
has been progressing with our own native storage (nioneo) as the primary
storage, which has made the rest of Neo4j more closely tied to that specific
implementation.

That said, there are still remnants of the time when the storage engine was
pluggable, so it could be possible to substitute your own engine. It is
going to take some patching of the internals and a fair bit of coding work,
but you are welcome to try it on your own.

If you don't mind me asking, why couldn't you just use Neo4j as is? That bit
was a bit lost in your message.

If I were you I would start out with writing a prototype of my application
using Neo4j as-is. That would validate the benefits of the Neo4j graph
database model, and show the benefits it would have to solving the problems
that the application is meant to solve. This prototype I would then bring to
my co-workers, showing them the benefits of using a graph for structuring
data. First then would I start thinking of how this would fit in with the
rest of the infrastructure for the application, does the benefits outweigh
the cost of adding another storage engine? If the value of it is high *and*
the cost of adding another storage engine is high, then I would consider
adapting the model to the existing storage, but carefully being aware of the
cost of doing so (i.e. losing the ability to get commercial support from the
vendor, Neo Technology). I'd start with quick simple prototypes to verify to
myself that the performance and other values of Neo4j didn't get lost too
much to make it unfit to my needs as I adapted the model to that other
storage solution.

I hope that was at least the beginning of an answer to your question. I'm
looking forward to your follow-up.

Cheers,
Tobias

On Sun, Apr 10, 2011 at 3:15 AM, elangovan shanmugam elangov...@hotmail.com
 wrote:


 Hi, I am new to Neo4j. Currently trying some simple graph prototypes to our
 business problems. Neo4j comes up with its own storage technology. Is it
 possible to use other storage technologies (say MongoDB) with Neo4j?
 The main reason is that... I will be if I asked for another storage
 technology in our group. I just convinced that we needed MongoDB, I would
 like to see if I can leverage that with Neo4j.
 Thanks  Appreciate your help.
 Elangovan S


 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Number of nodes/relationships visited in query?

2011-04-10 Thread Tobias Ivarsson
That information is not available at the moment. But it is an interesting
idea. How would you like to access it? Can you give a quick sketch of how
you envision the API for it.

Would it be something like this?

interface PathFinderP extends Path {
P findSinglePath( Node start, Node end ); // ok, nothing added here
// you'd have to use this version to get those statistics:
CountIterableP findAllPaths( Node start, Node end );
}

interface CountIterableT extends IterableT {
CountIteratorT iterator();
}

interface CountIteratorT extends IteratorT {
int numberOfNodesVisitedSoFar();
int numberOfRelationshipsVisitedSoFar();
}

(But with more thought through names)

Since computation isn't performed until actually requested (when the
iterator is iterated over), and since the Iterable could give a different
result when you iterate over it subsequent times (due to the graph being
modified), the Iterator object is the only object where I could see that
such information could be added usefully. This does mean that you cannot use
the java foreach loop with such an Iterable AND get the visited count, but
would have to resort to using the hasNext() and next() methods. We could
quite easily add some convenience methods for making that easier though,
something like:

CountIteratorPath pathIter = shortestPath.findAllPaths( start, end
).iterator();
for ( Path path : IterUtil.loop( pathIter ) ) {
doSomethingWith(path);
}
// after the loop is done, number of nodes visited so far is the total
number of visited nodes.
int visitedNodes = pathIter.numberOfNodesVisitedSoFar();

WDYT? If this is useful we could add this kind of statistics to all types of
traversals. Exposing that through the REST interface would be even simpler,
the implementation would simply do the equivalent of the iteration above
then add the statistics to the result. For paginated results (when those are
added) we could have the statistics reflect the number of nodes visited for
creating that page of data, for algorithms that find the easiest solutions
first, you could use those statistics to stop a search when the number of
nodes visited to collect a page grows too big.

Cheers,
Tobias

PS. I had a few other ideas on how such statistics information could be
added, but this was the least invasive of them. I sketched this up in five
minutes (after half a bottle of wine), so I'm very open to the possibility
of there being more elegant ways of adding this information.

On Sun, Apr 10, 2011 at 10:45 PM, Stephan Hagemann 
stephan.hagem...@googlemail.com wrote:

 Hi all,

 is there a way for me to get to the number of nodes or relationships that
 the graph algorithms (like ShortestPath) have visited and output it with
 the
 result of the query? I would like to do that to analyze the performance and
 load of query processing. Any ideas?

 Thanks!

 Stephan
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Trouble with Exclude and Continue in Traversal

2011-04-08 Thread Tobias Ivarsson
:632)
at
 sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195)
at
 java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
at
 com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:218)
at
 com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:129)
... 14 more

 What have I done wrong?

 Thanks Mattias for helping me doing lot of the code in the Traversal
 description.

 Brendan
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] CNFE in Neo4j Server

2011-04-07 Thread Tobias Ivarsson
The simplest explanation is probably the correct one: it is not on the
classpath.
This class is to be found in the jersey-server-1.3.jar is that jar on the
classpath? (or in jersey-client-1.3.jar, either will do)

The simplest way one might be fooled into not including it in the classpath
is due to there not being a compile-time dependency on it. the JAX-RS API
does not require an implementation for compiling, but at runtime you must
have a valid JAX-RS implementation available on the classpath. The default
implementation is the one defined by com.sun.ws.rs.ext.RuntimeDelegateImpl,
a class that jersey provides, being the reference implementation.

-tobias

On Thu, Apr 7, 2011 at 7:19 AM, Peter Neubauer 
peter.neuba...@neotechnology.com wrote:

 Hi there,
 trying to execute

 https://github.com/peterneubauer/neo4j-gremlin-plugin/blob/master/src/test/org/neo4j/server/plugin/gremlin/GremlinPluginTest.java
 as a JUnit test on Eclipse/Mac with default JDK, Saikat is getting the
 following:


 java.lang.ExceptionInInitializerError
   at
 org.neo4j.server.rest.repr.formats.JsonFormat.init(JsonFormat.java:43)
   at
 org.neo4j.server.plugin.gremlin.GremlinPluginTest.setUpBeforeClass(GremlinPluginTest.java:37)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at
 org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
   at
 org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
   at
 org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
   at
 org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
   at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
   at
 org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49)
   at
 org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
   at
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
   at
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
   at
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
   at
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
 Caused by: java.lang.RuntimeException:
 java.lang.ClassNotFoundException:
 com.sun.ws.rs.ext.RuntimeDelegateImpl
   at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:122)
   at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91)
   at javax.ws.rs.core.MediaType.clinit(MediaType.java:44)
   ... 17 more
 Caused by: java.lang.ClassNotFoundException:
 com.sun.ws.rs.ext.RuntimeDelegateImpl
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
   at java.lang.Class.forName0(Native Method)
   at java.lang.Class.forName(Class.java:169)
   at javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:62)
   at javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:155)
   at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:105)
   ... 19 more

 Any clues on why there is a class missing?

 Cheers,

 /peter neubauer

 GTalk:  neubauer.peter
 Skype   peter.neubauer
 Phone   +46 704 106975
 LinkedIn   http://www.linkedin.com/in/neubauer
 Twitter  http://twitter.com/peterneubauer

 http://www.neo4j.org   - Your high performance graph database.
 http://startupbootcamp.org/- Öresund - Innovation happens HERE.
 http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Constructing an evaluator that only takes specific nodes from a path

2011-04-07 Thread Tobias Ivarsson
That would be EXCLUDE_AND_PRUNE.

On Thu, Apr 7, 2011 at 7:58 AM, Dario Rexin dario.re...@xing.com wrote:

 Hi Peter, yes that would be not to include the path in the result set.



 On 07.04.2011, at 07:01, Peter Neubauer 
 peter.neuba...@neotechnology.com wrote:

  Dario,
  I am not quite sure I understand what you mean by drop is that not
  to include the path into the result or prune or something else? Do you
  have a concrete example of this, maybe a simple toy graph test with
  toy information?
 
  Cheers,
 
  /peter neubauer
 
  GTalk:  neubauer.peter
  Skype   peter.neubauer
  Phone   +46 704 106975
  LinkedIn   http://www.linkedin.com/in/neubauer
  Twitter  http://twitter.com/peterneubauer
 
  http://www.neo4j.org   - Your high performance graph
 database.
  http://startupbootcamp.org/- Öresund - Innovation happens HERE.
  http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.
 
 
 
  On Wed, Apr 6, 2011 at 6:35 PM, Dario Rexin dario.re...@xing.com
 wrote:
  Hello,
 
  we are trying to construct an Evaluator that will select only one node
 from the middle of a path and include it in the result. We do have a
 specific path we are looking for and the endNode is stored in 'end'. Now
 what we would like to do is have an Evaluator like this:
 
   import org.neo4j.graphdb.traversal.
  Evaluator;
 
   new Evaluator(){
 @Override
 public Evaluation evaluate(org.neo4j.graphdb.Path path) {
 if(path.length()  3) {
 return Evaluation.EXCLUDE_AND_PRUNE;
 } else if (path.length() == 2) {
 return Evaluation.INCLUDE_AND_CONTINUE;
 } else if (path.length()  2){
 return Evaluation.EXCLUDE_AND_CONTINUE;
 } else {
 return path.endNode().getId() == end.getId() ?
 Evaluation.EXCLUDE_AND_PRUNE : Evaluation.DROP_PATH_AND_PRUNE;
 }
 }
   }
 
  The decision on whether to accept or drop a path is based on wether we
 have reached the endNode. That's why we would like to use something like
 DROP_PATH_AND_PRUNE, which doesn't exist. What are we missing?
 
  Any help appreciated!
 
  Thanks,
  Stephan and Dario
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Lucene 3.1 with neo4j. Is it OK?

2011-04-07 Thread Tobias Ivarsson
I see no reason for it not to work, Lucene has a good story of compatibility
between releases with the same major version.

But please note that this isn't something Neo Technology would officially
support (yet), so from that point of view you'd be on your own.

-t

On Thu, Apr 7, 2011 at 11:16 AM, Balazs E. Pataki pat...@dsd.sztaki.huwrote:

 Hi,

 the subject says is all. :-) But anyway, is it possible/safe to use the
 brand new 3.1 (31 March 2011) release of Lucene with neo4j?

 Thanks,
 ---
 balazs
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] building from source

2011-04-07 Thread Tobias Ivarsson
The failures you are seeing are for tests that deal with large data. They
thus need a large heap configuration to succeed.

For specifying such settings for maven you would use the MAVEN_OPTS
environment variable, something like:

export MAVEN_OPTS=-Xmx1G

Cheers,
Tobias

On Thu, Apr 7, 2011 at 4:58 PM, Will Holcomb w...@dhappy.org wrote:

 Thanks. I was trying to use the jar target rather than install.

 That build fails with:

 ---
  T E S T S
 ---
 Running org.neo4j.com.TestCommunication
 Thu Apr 07 10:53:14 EDT 2011: MadeUpServer communication server started and
 bound to 1234
 [MadeUpServerProcess:27271] Thu Apr 07 10:53:15 EDT 2011: MadeUpServer
 communication server started and bound to 
 [MadeUpServerProcess:27271] Thu Apr 07 10:53:24 EDT 2011: MadeUpServer
 shutdown, closing all channels
 Tests run: 5, Failures: 1, Errors: 3, Skipped: 0, Time elapsed: 10.353 sec
  FAILURE!

 Results :

 Failed tests:
  clientGetResponseFromServerViaComLayer(org.neo4j.com.TestCommunication)

 Tests in error:
  makeSureClientStoreIdsMustMatch(org.neo4j.com.TestCommunication)
  makeSureServerStoreIdsMustMatch(org.neo4j.com.TestCommunication)
  makeSureClientCanStreamBigData(org.neo4j.com.TestCommunication)

 Tests run: 5, Failures: 1, Errors: 3, Skipped: 0


 I can get it to build though with: mvn -Dmaven.test.skip=true install

 –☮W

 On Thu, Apr 7, 2011 at 1:05 AM, Peter Neubauer 
 peter.neuba...@neotechnology.com wrote:

  Willi,
  the main build tool is maven, and we are in the middle of a
  major reshuffling on GIT, so if you install maven.apache.org and try
  mvn clean install that, things should work out better. Would that be
  an option?
 
  Cheers,
 
  /peter neubauer
 
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Lucene 3.1 with neo4j. Is it OK?

2011-04-07 Thread Tobias Ivarsson
1.3 is frozen, nothing new is going in until the 1.4 line.

-tobias

On Thu, Apr 7, 2011 at 6:32 PM, Rick Bullotta
rick.bullo...@thingworx.comwrote:

 Any plans to officially test Lucene 3.1 for Neo 1.3 GA?

 There are plenty of useful things and bug fixes in Lucene 3.1.


 -Original Message-
 From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org]
 On Behalf Of Tobias Ivarsson
 Sent: Thursday, April 07, 2011 5:45 AM
 To: Neo4j user discussions
 Subject: Re: [Neo4j] Lucene 3.1 with neo4j. Is it OK?

 I see no reason for it not to work, Lucene has a good story of
 compatibility
 between releases with the same major version.

 But please note that this isn't something Neo Technology would officially
 support (yet), so from that point of view you'd be on your own.

 -t

 On Thu, Apr 7, 2011 at 11:16 AM, Balazs E. Pataki pat...@dsd.sztaki.hu
 wrote:

  Hi,
 
  the subject says is all. :-) But anyway, is it possible/safe to use the
  brand new 3.1 (31 March 2011) release of Lucene with neo4j?
 
  Thanks,
  ---
  balazs
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 



 --
 Tobias Ivarsson tobias.ivars...@neotechnology.com
 Hacker, Neo Technology
 www.neotechnology.com
 Cellphone: +46 706 534857
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] InvalidRecordException

2011-04-06 Thread Tobias Ivarsson
This has been very thoroughly investigated, and still is being thoroughly
investigated, since we have not been able to reproduce the problem at all.
Any hints on what scenario this occurred in that you can provide would be
immensely helpful.

Cheers,
Tobias

On Tue, Apr 5, 2011 at 9:41 PM, Degrassi Francesco 
francesco.degra...@emaze.net wrote:

 Hello everyone.
 Was this issue investigated and, perhaps, solved ?
 We started experiencing the same today, on Neo 1.1 on CentOS 5 and couldn't
 manage to reproduce it reliably yet.

 I will attach a stacktrace and more info tomorrow, but wanted to know if
 there
 were any news on the issue.

 Thanks in advance

 Francesco Degrassi

 Il 10 marzo 2011 alle 21.58 Massimo Lusetti mluse...@gmail.com ha
 scritto:

  On Thu, Mar 10, 2011 at 6:11 PM, Axel Morgner a...@morgner.de wrote:
 
   Hi,
  
   I'm getting an InvalidRecordException
  
   org.neo4j.kernel.impl.nioneo.store.InvalidRecordException: Node[5] is
   neither firstNode[37781] nor secondNode[37782] for Relationship[188125]
   at
  
 org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:131)
   at
  
 org.neo4j.kernel.impl.nioneo.xa.NioNeoDbPersistenceSource$ReadOnlyResourceConnection.getMoreRelationships(NioNeoDbPersistenceSource.java:280)
   at
  
 org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:100)
   at
  
 org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:585)
   at
  
 org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:358)
   at
  
 org.neo4j.kernel.impl.core.IntArrayIterator.hasNext(IntArrayIterator.java:115)
  
   when iterating through the relationships of a certain node:
  
   Node node = graphDb.getNodeById(sNode.getId());
  
   IterableRelationship rels =
   node.getRelationships(relType, dir);
  
   for (Relationship r : rels) { - here the expeption
   occurs
   ...
   }
  
   I'm using 1.3.M03.
  
   Seems that the database is in an inconsitant state. Don't know how this
   could happen ...
  
  
   Greetings
  
   Axel
  
   ___
   Neo4j mailing list
   User@lists.neo4j.org
   https://lists.neo4j.org/mailman/listinfo/user
  
 
 
  I'm encountering this same/similar exception quite ofter lately, with
  the same 1.3.M03 version, on FreeBSD 8.2 with  OpenJDK Runtime
  Environment (build 1.6.0-b21) OpenJDK 64-Bit Server VM (build
  19.0-b09, mixed mode):
 
  org.neo4j.kernel.impl.nioneo.store.InvalidRecordException:
  Record[4130751] not in use
  at

 org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java:194)
  at

 org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java:96)
  at

 org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.connectRelationship(WriteTransaction.java:1435)
  at

 org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.relationshipCreate(WriteTransaction.java:1389)
  at

 org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaConnection$RelationshipEventConsumerImpl.createRelationship(NeoStoreXaConnection.java:256)
  at

 org.neo4j.kernel.impl.nioneo.xa.NioNeoDbPersistenceSource$NioNeoDbResourceConnection.relationshipCreate(NioNeoDbPersistenceSource.java:370)
  at

 org.neo4j.kernel.impl.persistence.PersistenceManager.relationshipCreate(PersistenceManager.java:153)
  at

 org.neo4j.kernel.impl.core.NodeManager.createRelationship(NodeManager.java:309)
  at

 org.neo4j.kernel.impl.core.NodeImpl.createRelationshipTo(NodeImpl.java:387)
  at

 org.neo4j.kernel.impl.core.NodeProxy.createRelationshipTo(NodeProxy.java:186)
 
  Did it rings and alert bell!?
 
  Cheers
  --
  Massimo
  http://meridio.blogspot.com
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] EmbeddedReadOnlyGraphDatabase workings

2011-04-06 Thread Tobias Ivarsson
/mailman/listinfo/user
 
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Query paths of certain lengths...

2011-03-31 Thread Tobias Ivarsson
Hi Jeff,

What you are describing is a Friends-of-a-friend (foaf) traversal, which
are the friends of my friends, who are not my direct friends.

The best way to perform such a traversal is by using the traversal API:

n.traverse( Order.BREADTH_FIRST, // because we want to exclude the direct
friends early
  new StopEvaluator() { // don't traverse deeper than depth two
  boolean isStopNode(TraversalPosition pos) { return pos.depth() == 2; }
   }, new ReturnableEvaluator() { // return the ones found on depth two
  boolean isReturnableNode(TraversalPosition pos) { return pos.depth()
== 2; }
   }, RelTypes.FRIEND, Direction.BOTH ); // Traverse Friend relationships,
in any direction

Since this traverses breadth first, it will find the direct friends first,
and exclude those, since this kind of traversal does not revisit nodes.

The traversal above almost does what you want, except that it only returns
nodes, and you wanted the paths. For that you can use our more capable
tentative traversal API to do the same traversal, but getting paths instead:

TraversalDescription foaf = Traversal.description()
.breadthFirst()
.relationships( RelTypes.FRIEND, Direction.OUTGOING )
.evaluator( new Evaluator() {
Evaluation evaluate( Path path )
{
return ( path.length() == 2 ) ? Evaluation.INCLUDE_AND_PRUNE :
Evaluation.EXCLUDE_AND_CONTINUE;
}
} );

Then you'd just iterate over the paths from the given node:

for (Path foafPath : foaf.traverse( n ) ) {
printPath( foafPath );
}

If you want that for ALL possible start-nodes, I guess you would have to
iterate through all nodes and apply the same traversal. But given just a
single start node, the above traversal is enough, and it is fast enough to
do in real time (show the friends of my friends on a social media website).

Cheers,
Tobias

On Wed, Mar 30, 2011 at 7:51 PM, jisenhart jisenh...@yoholla.com wrote:


 Suppose I have the following node/paths

 n - n1
 n - n2
 n - n4

 n2 - n1
 n2 - n3
 n2 - n4

 I want to find all paths of depth two (for example):

 n - n2 - n3
 n - n2 - n4

 and filter out those paths where a shorter path exists to a given node
 (n) leaving just

 n - n2 - n3

 since n - n4 is shallower than n - n2 - n4


 Is this possible? I see GraphAlgoFactory.pathsWithLength(expander,
 length). But unclear on how to proceed beyond that.

 Jeff

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Glassfish problem

2011-03-27 Thread Tobias Ivarsson
Hi Matěj,

As the error message says, most likely this is caused by another instance
already being open in the same JVM.

If I were to have I guess, I think that your code (in the singleton you
mention) create a new GraphDatabaseService without invoking shutdown() on
the existing one first. Probably what you've done is just null out the
reference (which is why your code thinks it has to start a new instance).

There is no need to startup and shutdown the graph database multiple times
(are you doing this for each request?), it is not a database connection, it
is the database, and startup and shutdown takes time. It is better to tie
the lifecycle of the graph database to the lifecycle of your application.

Cheers,
Tobias

2011/3/28 Matěj Plch plchm...@fit.cvut.cz

 Hi! So we deployed our application, but we get Caused by:
 java.lang.IllegalStateException: Unable to lock store ... SEVERE:
 0-database/neostore], this is usually caused by another Neo4j kernel
 already running in this JVM for this particular store
 Is there any best practise how to avoid this? The database is opened or
 created from class with is Singleton. Can this be the problem?
 I googled this problem, but no solution found.
 Matěj Plch
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Filter by relationship Property

2011-03-24 Thread Tobias Ivarsson
For the first position (at the start node), there is no relationship leading
up to it, which means that the lastRelationship() method returns null.
This is a special case that has to be dealt with.

Cheers,
Tobias

On Wed, Mar 23, 2011 at 9:05 PM, Justine Mathews justine.math...@megree.com
 wrote:

 Tobias,



 I have used the return filter, I am getting following error when making
 rest call for node  traverse,

 Error : The remote server returned an error: (500) Internal Server Error.

 Response header status : *javax.script.ScriptException:
 sun.org.mozilla.javascript.internal.EcmaError: TypeError: Cannot call method
 getProperty of null (Unknown Source#1)*



 please find below the syntax,



 *url:* http://localhost:7474/db/data/node/80/traverse/node

 *traverse:*

 {order: depth first,relationships: [

   {

 type: Friend,

 direction: all

   }

 ],return filter: {

   language: javascript,

   body: position.lastRelationship().hasProperty(\customerId\);

 },max depth: 1}



 · *Kernel version*

 Neo4j - Graph Database Kernel 1.3.M03





 --
 Regards,
 Justine K Mathews, MCSD.NET
 Mob: +44-(0) 7795268546

 http://www.justinemathews.com

 http://uk.linkedin.com/in/justinemathews






-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] A picture worth a 170'000 nodes and 650'000 edges

2011-03-24 Thread Tobias Ivarsson
Nice to see my visualization code being put to good use.

The most recent version has moved to github, along with the rest of Neo4j,
since this is an unofficial tool it is on my personal github:
https://github.com/thobe/neoviz

Cheers,
Tobias

On Thu, Mar 24, 2011 at 8:09 PM, Alfredas Chmieliauskas al.fre...@gmail.com
 wrote:

 For fun:

 This is a visualization of the graph resulting from a simulation
 (interacting energy markets).

 http://test.eeni.tbm.tudelft.nl/~alfredas/d13n-graph.png

 After 10 simulation ticks we have ~170'000 entities (agents, markets,
 power plants, bids, substances, technologies, etc) and ~650'000
 relations between these entities. Different colors represent different
 types of relations. These relations are created at a rate of ~60'000
 per tick, as a result of agents trading, investing and energy flows.
 By the end of the simulation we get ~1m nodes and ~4m edges.

 The visualization was made using neo-graphviz:
 https://svn.neo4j.org/laboratory/components/neo-graphviz/

 The crazy part is that we can still make sense of what's happening
 there, using a few pipes and traversals :-)

 Alfredas
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Filter by relationship Property

2011-03-23 Thread Tobias Ivarsson
Hi Justine,

What you have specified is to prune traversal branches where customerId is
2, this means that the traversal will not continue further from that point,
but it doesn't mean that this position will not be returned. For that you
need to specify a return filter instead of or in addition to the prune
evaluator.

The syntax for return filter is the same as for prune evaluator, see:
http://components.neo4j.org/neo4j-server/stable/rest.html#Traverse

Cheers,
Tobias

On Tue, Mar 22, 2011 at 11:55 PM, Justine Mathews 
justine.math...@megree.com wrote:


 Please find below the traversal syntax I am using from rest client, The
 result is not filtered by the
 customerId( relationship property)  value 2.

 {order: depth first,relationships: [
  {
type: Friend,
direction: all
  }
 ],prune evaluator: {
  language: javascript,
  body: position.lastRelationship().getProperty(\customerId\) == 2;
 }}



 --
 Regards,
 Justine K Mathews, MCSD.NET
 Mob: +44-(0) 7795268546
 http://www.justinemathews.comhttp://www.justinemathews.com/
 http://uk.linkedin.com/in/justinemathews

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] neo4j reade with getNodeById index.with...

2011-03-23 Thread Tobias Ivarsson
It is correct that getNodeById is much faster than an index lookup, but IDs
are assigned by Neo4j, there is no way for you as a user to assign IDs,
which makes it a very blunt tool for looking up entities. To know which id
corresponds to a particular name or similar attribute, you would have to go
through an index, and that is exactly what the Neo4j index API does for you.

By the way, the LuceneIndexService is a deprecated API, use
GraphDatabaseService#index() instead:
http://components.neo4j.org/neo4j/1.3.M04/apidocs/org/neo4j/graphdb/GraphDatabaseService.html#index()

Cheers,
Tobias

On Wed, Mar 23, 2011 at 3:45 AM, 孤竹 ho...@foxmail.com wrote:

 HI,all

   I Found something interesting, The test as  follow:

 public static void main(String args[]) {
GraphDatabaseService db = new EmbeddedGraphDatabase(testDB);
indexService = new LuceneIndexService(db);
Long startTime = System.currentTimeMillis();
for(int i =0 ;i 1;i++){
Node node = getNodeByIndexProperty(db,String.valueOf(i));
}
Long endTime = System.currentTimeMillis();
System.out.println(time =  + (endTime - startTime));
}
public static Node getNodeByIndexProperty(GraphDatabaseService db,
 String keyValue) {
 //Node node = indexService.getSingleNode(KEY_NAME, keyValue);
Node node = db.getNodeById(Long.valueOf(keyValue));
return node;
}

 When I try to use the method indexService.getSingleNode.. with 
 times, It will take 2 seconds (nearly) , BY when I use db.getNodeById
 same times, It just take 350 millisecond . Does it mean getById is faster
 than use index ? or Does that mean the natural index is better than index if
 I can use it ?

 thanks for your help ,and replies, that help me very much!
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] 回复: neo4j reade with getNodeById index.with...

2011-03-23 Thread Tobias Ivarsson
Why not just keep the reference to the actual Node object then? Neo4j's
internal cache and memory management makes this really cheap to do. [1]

As to your questions about size of property of an index and maximum memory,
you will have to be clearer, I really don't understand what you are asking
for.

-tobias

[1] a Node object always occupies exactly 32B in a 64bit JVM, and I believe
that would translate to 20B on a 32bit JVM. The actual state of the Node is
stored internally in Neo4j and managed through the Neo4j cache.

2011/3/23 孤竹 ho...@foxmail.com

 Yes, I know I can't assign the IDS. But I can got some node's ID ,and cache
 it in some place( i.e. in the cache).


 when I need to search something , I got the id in the cache . Is it better
 ?


 At last, I wonder what size is property of the index? Depends On the memory
 I gived to Neo4j ?


 Is there a suggest rate ? for example , If the max-memory for Neo4j is 1G,
 and the index is better less than 500M ?


 -- 原始邮件 --
 发件人: mattiasmatt...@neotechnology.com;
 发送时间: 2011年3月23日(星期三) 下午4:24
 收件人: Neo4j user discussionsuser@lists.neo4j.org;

 主题: Re: [Neo4j] neo4j reade with getNodeById  index.with...


  And also to circumvent this different in performance for not-very-big
 indexes you can make use of the built-in caching (for the new API)... see

 http://docs.neo4j.org/chunked/snapshot/indexing-lucene-extras.html#indexing-lucene-caching

 2011/3/23 Tobias Ivarsson tobias.ivars...@neotechnology.com

  It is correct that getNodeById is much faster than an index lookup, but
 IDs
  are assigned by Neo4j, there is no way for you as a user to assign IDs,
  which makes it a very blunt tool for looking up entities. To know which
 id
  corresponds to a particular name or similar attribute, you would have to
 go
  through an index, and that is exactly what the Neo4j index API does for
  you.
 
  By the way, the LuceneIndexService is a deprecated API, use
  GraphDatabaseService#index() instead:
 
 
 http://components.neo4j.org/neo4j/1.3.M04/apidocs/org/neo4j/graphdb/GraphDatabaseService.html#index()
 
  Cheers,
  Tobias
 
  On Wed, Mar 23, 2011 at 3:45 AM, 孤竹 ho...@foxmail.com wrote:
 
   HI,all
  
 I Found something interesting, The test as  follow:
  
   public static void main(String args[]) {
  GraphDatabaseService db = new EmbeddedGraphDatabase(testDB);
  indexService = new LuceneIndexService(db);
  Long startTime = System.currentTimeMillis();
  for(int i =0 ;i 1;i++){
  Node node = getNodeByIndexProperty(db,String.valueOf(i));
  }
  Long endTime = System.currentTimeMillis();
  System.out.println(time =  + (endTime - startTime));
  }
  public static Node getNodeByIndexProperty(GraphDatabaseService db,
   String keyValue) {
   //Node node = indexService.getSingleNode(KEY_NAME, keyValue);
  Node node = db.getNodeById(Long.valueOf(keyValue));
  return node;
  }
  
   When I try to use the method indexService.getSingleNode.. with 
   times, It will take 2 seconds (nearly) , BY when I use
 db.getNodeById
   same times, It just take 350 millisecond . Does it mean getById is
 faster
   than use index ? or Does that mean the natural index is better than
 index
  if
   I can use it ?
  
   thanks for your help ,and replies, that help me very much!
   ___
   Neo4j mailing list
   User@lists.neo4j.org
   https://lists.neo4j.org/mailman/listinfo/user
  
 
 
 
  --
  Tobias Ivarsson tobias.ivars...@neotechnology.com
  Hacker, Neo Technology
  www.neotechnology.com
  Cellphone: +46 706 534857
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 



 --
 Mattias Persson, [matt...@neotechnology.com]
 Hacker, Neo Technology
 www.neotechnology.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Fans of Neo4j From Chinese

2011-03-22 Thread Tobias Ivarsson
Rick, I don't quite understand what you are asking for. Could you elaborate
further?

-tobias

2011/3/22 Rick Bullotta rick.bullo...@thingworx.com

 I'd like to explore this question a bit further.  Does this mean that
 basically there's no way to scale beyond a single thread/CPU for
 disconnected graphs if you have complex graph dependencies (e.g. you cannot
 create disjoint subgraphs)?



 -Original Message-
 From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org]
 On Behalf Of Tobias Ivarsson
 Sent: Saturday, March 19, 2011 5:59 AM
 To: Neo4j user discussions
 Subject: Re: [Neo4j] Fans of Neo4j From Chinese

 Neo4j serializes commits. I.e. at most one thread is committing a
 transaction at once.
 For the actual work of building up the data to be committed, Neo4j supports
 multiple concurrent threads.

 This fact alone, that there is a single congestion point, means that if an
 application, like in your case, is very write centric, it is unlikely for
 it
 to scale beyond two threads, with one building up the next commit while the
 other is commiting its data. It might scale to a few more threads than that
 if the buildup time is significantly larger than the commit time. It is
 simple time slicing, only one train can be at the station at once, then
 you have to do the maths on how many trains can be out on the track
 during
 that time.

 It is also worth keeping in mind, that for CPU bound operation, an
 application doesn't scale much further than the number of CPUs in the
 computer. The threads that are not in commit mode - i.e. the ones that are
 building up the data for their next commit - are CPU bound, and contending
 for the same CPU resources. This means that your application is not going
 to
 scale much further than the number of CPUs in your computer, and few
 desktop/laptop computers have more than 4 CPUs these days, which makes 5
 threads about the most you can squeeze out of it, anything more than that
 is
 just going to add contention, and possibly even slow things down.

 Finally, the (CPU bound) threads that create the graph might be contending
 on the same resources. As Peter said. If multiple threads modify the same
 node or relationship, i.e. if they create relationships to the same node
 (the root node for example), they are all going to block on that resource.
 Neo4j only allows one transaction to modify each entity at a time. This
 means that to get maximum concurrency out of your data creation, each
 thread
 should be creating each own disconnected subgraph. And if they have
 connected parts, the connections to the global data should be made last
 in
 the transaction (in a predictable order to avoid deadlocks[1]), to maximize
 the time the thread is operational before hitting the
 congestion point that is the (potentially) contended data.

 Cheers,
 Tobias

 [1] Neo4j will detect if a deadlock has occurred and throw a
 DeadlockDetectedException in that case.

 2011/3/18 孤竹 ho...@foxmail.com

  hi,
 
 
Sorry for disturb you , I am a chinese engineer , Excused for my bad
  english :) .
 
 
Recently, I am learning Neo4j and trying to use it in my project . But
  When I make a Pressure on neo4j with 5 theads , 10 theads, 20 and 30, I
  found the nodes inserted to the Neo4J is not change obvious (sometimes
 not
  change ~ ~! ). Does it not matter with threads ? the kenerl will make it
  Serial ? Is there any documents or something about The performance of
 Neo4j
  ? thanks for your help
 
 
 
The program as follows:
I put this function in ExecutorService ,with 5/10/30 threads. then test
  for the nodes inserted into at same time .(The counts have not changed
  obviously)
 
 
  Transaction tx = null;
 Node before = null;
 try {
 for (int i = 0; i  100; i++) {
 if(stop == true){
 return;
 }
 if (graphDb == null) {
 return;
 }
 try {
 if (tx == null) {
 tx = graphDb.beginTx();
 }
 // 引用计数加1
 writeCount.addAndGet(1);
 int startNodeString =
  name.addAndGet(1);
 Node start =
  getOrCreateNodeWithOutIndex(
 +
 startNodeString);
 if (before == null) {
 // 根节点.哈哈哈 I got U
 Node root =
  graphDb.getNodeById(0);
 
   root.createRelationshipTo(start, LEAD

Re: [Neo4j] Fans of Neo4j From Chinese

2011-03-19 Thread Tobias Ivarsson
();
tx = null;
}
} catch (Exception e) {
 System.out.println(write : =  +
 e);
}
}
} catch (Exception e) {
} finally {
tx.finish();
}
}

-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Problems installing neo4j.py on Mac

2011-03-17 Thread Tobias Ivarsson
/examples/python/djangosites/faces/views.py
 Aneo4j-python/src/examples/python/djangosites/faces/__init__.py
 Aneo4j-python/src/examples/python/djangosites/faces/tests.py
 Aneo4j-python/src/examples/python/djangosites/faces/models.py
 Aneo4j-python/src/examples/python/djangosites/faces/urls.py
 Aneo4j-python/src/examples/python/djangosites/blog
 Aneo4j-python/src/examples/python/djangosites/blog/views.py
 Aneo4j-python/src/examples/python/djangosites/blog/__init__.py
 Aneo4j-python/src/examples/python/djangosites/blog/tests.py
 Aneo4j-python/src/examples/python/djangosites/blog/models.py
 Aneo4j-python/src/examples/python/djangosites/blog/urls.py
 Aneo4j-python/src/examples/python/djangosites/__init__.py
 Aneo4j-python/src/examples/python/djangosites/settings.py
 Aneo4j-python/src/examples/python/djangosites/imdb
 Aneo4j-python/src/examples/python/djangosites/imdb/orm_model.py
 Aneo4j-python/src/examples/python/djangosites/imdb/views.py
 Aneo4j-python/src/examples/python/djangosites/imdb/__init__.py
 Aneo4j-python/src/examples/python/djangosites/imdb/importer.py
 Aneo4j-python/src/examples/python/djangosites/imdb/neo_model.py
 Aneo4j-python/src/examples/python/djangosites/imdb/data
 A
  neo4j-python/src/examples/python/djangosites/imdb/data/test-actors.list
 Aneo4j-python/src/examples/python/djangosites/imdb/data/sample.py
 Aneo4j-python/src/examples/python/djangosites/imdb/data/actors.txt
 A
  neo4j-python/src/examples/python/djangosites/imdb/data/test-movies.list
 Aneo4j-python/src/examples/python/djangosites/imdb/tests.py
 Aneo4j-python/src/examples/python/djangosites/imdb/models.py
 Aneo4j-python/src/examples/python/djangosites/imdb/templatetags
 A
  neo4j-python/src/examples/python/djangosites/imdb/templatetags/__init__.py
 A
  neo4j-python/src/examples/python/djangosites/imdb/templatetags/apply.py
 Aneo4j-python/src/examples/python/djangosites/imdb/urls.py
 Aneo4j-python/src/examples/python/djangosites/urls.py
 Aneo4j-python/src/examples/python/djangosites/templates
 A
  neo4j-python/src/examples/python/djangosites/templates/list_of_links.html
 Aneo4j-python/src/examples/python/djangosites/templates/imdb
 A
  neo4j-python/src/examples/python/djangosites/templates/imdb/parts_table.html
 A
  neo4j-python/src/examples/python/djangosites/templates/imdb/list_view.html
 Aneo4j-python/src/examples/python/djangosites/templates/imdb/index.html
 A
  neo4j-python/src/examples/python/djangosites/templates/imdb/searchform.html
 Aneo4j-python/src/examples/python/djangosites/templates/imdb/view.html
 Aneo4j-python/src/examples/python/djangosites/templates/imdb/list.html
 Aneo4j-python/src/examples/python/djangosites/templates/base.html
 Aneo4j-python/pom.xml
 Aneo4j-python/README.txt
 Aneo4j-python/tech-debt.txt
 Exported revision 8590.

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Neo4j OSGI bundle

2011-03-12 Thread Tobias Ivarsson
Oh, right.

Only the bundles that org.neo4j:neo4j depends on exports packages. I
guess org.neo4j:neo4j
would have to re-export the packages it depends on, so that your application
can depend on org.neo4j:neo4j and use those packages. Is that the
appropriate way? I have little OSGi experience (other than cursing over it
when it fails the build process because you need to very explicitly declare
which packages to include).

If that would solve things for you we will make sure to make those changes
for the next milestone. In the meantime you can directly depend on the
bundles org.neo4j:neo4j depends on, you are probably interested in at least:
org.neo4j:neo4j-kernel
org.neo4j:neo4j-lucene-index
But perhaps these as well:
org.neo4j:neo4j-management - JMX support
org.neo4j:neo4j-graph-algo - Graph algorithms, such as shortest path
org.neo4j:neo4j-com - Communication support, this includes the online-backup
server, and the bases that HA depends on
org.neo4j:neo4j-ha - High availability support for Neo4j
You might also want to be nice to us and have Neo4j send a ping to us every
day, letting us know how many people use Neo4j:
org.neo4j:neo4j-udc

Please let us know how this works out for you, and if there are any
problems. We have only done casual testing with OSGi, so any feedback is
much appreciated.

Cheers,
Tobias

On Fri, Mar 11, 2011 at 10:40 PM, Nicolas Jouanin nicolas.joua...@gmail.com
 wrote:

 Hi,

 I'm planning to use Neo4j under OSGi framework. Because org.neo4j:neo4j
 defines the needed headers (in MANIFEST.MF), it can be deployed as a bundle
 on equinox or felix.
 Unfortunatelly, the bundle doesn't export any package so it's quite
 unuseful for me as my own bundle can't import them.
 I guess, at least neo4j bundle should have Export-Package directive in
 MANIFEST.MF file.
 Do you think these headers could be added in the next snapshot ?

 Thanks.

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] InvalidRecordException

2011-03-11 Thread Tobias Ivarsson
Please tell me that you have the database file in the state they were in
when this happened. That you have not tried to repair the database in any
way.
If you do, could you please send me those database files off-list (dropbox
or similar, or contact me directly to discuss how to transfer it).

I will of course treat your data with utmost privacy, but I would love to
get to examine a store that exhibits this behavior to find out what could
have caused it.

Massimo: If you can produce a test case that reliably (or at least
semi-reliably) reproduces this, that would of course be even better, but it
doesn't hurt if I could look at the store files and try to deduce the
problem in parallell.

Cheers,
-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] InvalidRecordException

2011-03-11 Thread Tobias Ivarsson
I need at least these two files:
neostore.nodestore.db
neostore.relationshipstore.db

-tobias

On Fri, Mar 11, 2011 at 11:26 AM, Massimo Lusetti mluse...@gmail.comwrote:

 On Fri, Mar 11, 2011 at 10:49 AM, Tobias Ivarsson
 tobias.ivars...@neotechnology.com wrote:

  Please tell me that you have the database file in the state they were in
  when this happened. That you have not tried to repair the database in any
  way.
  If you do, could you please send me those database files off-list
 (dropbox
  or similar, or contact me directly to discuss how to transfer it).

 Sure I have it unmodified and could make it available to you with
 dropbox but the db dir it's 2G... so is there something I could leave
 out?
 If you prefer you can continue to mail privately...

  Massimo: If you can produce a test case that reliably (or at least
  semi-reliably) reproduces this, that would of course be even better, but
 it
  doesn't hurt if I could look at the store files and try to deduce the
  problem in parallell.

 I'm working on it... will let you know soon.

 Thanks for the interest... Cheers
 --
 Massimo
 http://meridio.blogspot.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Announcing Neo4j 1.3 “Abisko Lampa” M04

2011-03-11 Thread Tobias Ivarsson
It is in git: https://github.com/neo4j/graphdb

Neo4j moved from svn to git a little over a month ago.

Cheers,
Tobias

On Fri, Mar 11, 2011 at 3:21 PM, Jonah H. Harris 
jonah.har...@myyearbook.com wrote:

 On Fri, Mar 11, 2011 at 9:02 AM, Chris Gioran 
 chris.gio...@neotechnology.com wrote:

  The most important feature of this release is of course the expansion of
  the

 storage layer to allow for even more entities,


 Hey Chris,

 This sounds great.  Is it checked into SVN?  I'd like to see the
 differences
 in the file formats.

 --
 Jonah H. Harris, VP of Database Administration
 myYearbook.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] How to copy a complete database?

2011-03-07 Thread Tobias Ivarsson
That class is in Neo4j kernel. It is the actual implementation class.
You can find the source code here:
https://github.com/neo4j/graphdb/blob/master/kernel/src/main/java/org/neo4j/kernel/impl/nioneo/store/ShortString.java

All of the classes in the org.neo4j.kernel.impl.nioneo.store package of my
neo4j-admin-store project are of that kind, facade classes that provide
access to internals of the neo4j kernel implementation.

Cheers,
Tobias

On Mon, Mar 7, 2011 at 1:00 PM, Balazs E. Pataki pat...@dsd.sztaki.huwrote:

 Hi Tobias,

 I tried to use ShortStringEncoding.java, however it seems to refer to a
 class called ShortString, which is not available in github:


 https://github.com/thobe/neo4j-admin-store/tree/master/src/main/java/org/neo4j/kernel/impl/nioneo/store

 Could you also please provide it?

 Thanks,
 ---
 balazs

 On 3/4/11 5:05 PM, Tobias Ivarsson wrote:
  Hi Balazs,
 
  I have created such a thing, the tool is available here:
 
 https://github.com/downloads/thobe/neo4j-admin-store/propstat-1.3.M03.jar
  Run it as: java -jar propstat-1.3.M03.jarpath to your neo4j store
 
  This tool will gather statistics about all of your properties, and output
  how many you have of each kind.
  For short string properties it will output how many you have in each
  encoding.
  For regular string properties it will output how many could have been
 stored
  as short strings, and with which encoding that would be.
 
  That jar file contains a class that can be used for testing if a given
  string is short or not, and how it would be stored if it was considered
  short:
 
 https://github.com/thobe/neo4j-admin-store/blob/master/src/main/java/org/neo4j/kernel/impl/nioneo/store/ShortStringEncoding.java
  The method you are interested in is (the first one) getEncoding(String),
 it
  returns one of the Enum constants in that class, or null if the provided
  string would not be considered short.
 
  While I wrote this I realized that it would be really simple to write a
 tool
  that converts all strings that are short enough to be stored as short
  strings, that tool is available here:
 
 https://github.com/downloads/thobe/neo4j-admin-store/makeshort-1.3.M03.jar
  Run it as: java -jar makeshort-1.3.M03.jarpath to your neo4j store
 
  This tool will modify your string properties to store them as short
 strings
  when possible. It will then rebuild the id generator for the
  DynamicStringStore so that the holes created where the strings were
  previously stored will be reused the next time you write string
 properties
  to Neo4j (unless they are also short of course).
 
  Please note that both of these tools come with no official support or
  guarantees. Use at your own risk.
 
  Cheers,
  Tobias
 
  On Fri, Mar 4, 2011 at 10:19 AM, Balazs E. Patakipat...@dsd.sztaki.hu
 wrote:
 
  Actually both options would be very useful, but the first option (string
  as input and tell me if it is short or not) would be enough for me for
  now if available.
 
  Thanks,
  ---
  balazs
 
  On 3/3/11 6:37 PM, Tobias Ivarsson wrote:
  Yes, the statistics tool almost does that, but with a wider range of
  possible encodings.
 
  What would you want from a class like that? Give it a string as input
 and
  have it tell you if it is short or not? Or go through an existing store
  and
  tell you how many of the strings are short?
 
  -tobias
 
  On Thu, Mar 3, 2011 at 5:18 PM, Balazs E. Patakipat...@dsd.sztaki.hu
  wrote:
 
  Hi Tobias,
 
  Do you happen to have some little class which can at tell us which
  property values neo4j would consider short and which of your 6
 encoding
  would it use? Maybe the string statistics collecting program you
 created
  earlier has this logic - but I'm just guessing.
 
  Thanks,
  ---
  balazs
 
  On 3/3/11 11:52 AM, Tobias Ivarsson wrote:
  No there is no simpler way, yet. We've been thinking about creating a
  short
  string compression tool for accomplishing this, but haven't done so
  yet.
 
  Cheers,
  Tobias
 
  On Thu, Mar 3, 2011 at 11:35 AM, Balazs E. Pataki
 pat...@dsd.sztaki.hu
  wrote:
 
  Hi,
 
  I have a big database based on Neo4J 1.2. Now, if I would like to
 use
  the short strings feature of Neo4j 1.3 M03 I should regenerate my
  full
  database, that is all strings should be reset so that it may or may
  not
  be stored according to the new short strings policy.
 
  It seems to me that the easiest way to do this would be to somehow
 be
  able to copy the full 1.2 database to a newly created 1.3 M03
 database
  by traversing the 1.2 database. But there maybe a simpler (neo4j
  builtin) way to do this. Any hints about this?
 
  Thanks,
  ---
  balazs
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 
 
 
 
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Slow down on insertion as db grow

2011-03-03 Thread Tobias Ivarsson
Hi Massimo,

I'm terribly sorry, I've had a lot of things on my plate, and this sadly
fell off. Thanks for the reminder!

Could you please try lowering the heap size (to somewhere between 512M-1G
with your available RAM). The Neo4j memory mapping is managed outside of the
Java heap. What you want to achieve for ideal performance is to get the
entire nodestore and relationshipstore to fit in the memory mapped file
cache. You might have to tweak the memory mapping settings to make sure that
they are configured to prefer the nodestore and relationship, and have them
fully memory mapped. See documentation here:
http://docs.neo4j.org/chunked/snapshot/configuration-caches.html#_configuration

I didn't see any information on which operating system you are using, so I
assumed linux or some other unix. If you are running on Windows the memory
mapping IS managed in the Java heap, then you should keep your heap settings
at the same size (1.5G) and instead just focus on configuring the memory
mapping so that the nodestore and relationship store are fully memory
mapped.

Cheers,
Tobias

On Thu, Mar 3, 2011 at 11:18 AM, Massimo Lusetti mluse...@gmail.com wrote:

 On Fri, Feb 18, 2011 at 2:31 PM, Tobias Ivarsson
 tobias.ivars...@neotechnology.com wrote:

  WRT the slowdown you've seen, I'll have to investigate that further.

 Didn't mean to bother anyone but do you have any news regarding the
 slowdown I'm still experiencing... !?

 Cheers
 --
 Massimo
 http://meridio.blogspot.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] How to copy a complete database?

2011-03-03 Thread Tobias Ivarsson
No there is no simpler way, yet. We've been thinking about creating a short
string compression tool for accomplishing this, but haven't done so yet.

Cheers,
Tobias

On Thu, Mar 3, 2011 at 11:35 AM, Balazs E. Pataki pat...@dsd.sztaki.huwrote:

 Hi,

 I have a big database based on Neo4J 1.2. Now, if I would like to use
 the short strings feature of Neo4j 1.3 M03 I should regenerate my full
 database, that is all strings should be reset so that it may or may not
 be stored according to the new short strings policy.

 It seems to me that the easiest way to do this would be to somehow be
 able to copy the full 1.2 database to a newly created 1.3 M03 database
 by traversing the 1.2 database. But there maybe a simpler (neo4j
 builtin) way to do this. Any hints about this?

 Thanks,
 ---
 balazs
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] How to copy a complete database?

2011-03-03 Thread Tobias Ivarsson
Yes, the statistics tool almost does that, but with a wider range of
possible encodings.

What would you want from a class like that? Give it a string as input and
have it tell you if it is short or not? Or go through an existing store and
tell you how many of the strings are short?

-tobias

On Thu, Mar 3, 2011 at 5:18 PM, Balazs E. Pataki pat...@dsd.sztaki.huwrote:

 Hi Tobias,

 Do you happen to have some little class which can at tell us which
 property values neo4j would consider short and which of your 6 encoding
 would it use? Maybe the string statistics collecting program you created
 earlier has this logic - but I'm just guessing.

 Thanks,
 ---
 balazs

 On 3/3/11 11:52 AM, Tobias Ivarsson wrote:
  No there is no simpler way, yet. We've been thinking about creating a
 short
  string compression tool for accomplishing this, but haven't done so yet.
 
  Cheers,
  Tobias
 
  On Thu, Mar 3, 2011 at 11:35 AM, Balazs E. Patakipat...@dsd.sztaki.hu
 wrote:
 
  Hi,
 
  I have a big database based on Neo4J 1.2. Now, if I would like to use
  the short strings feature of Neo4j 1.3 M03 I should regenerate my full
  database, that is all strings should be reset so that it may or may not
  be stored according to the new short strings policy.
 
  It seems to me that the easiest way to do this would be to somehow be
  able to copy the full 1.2 database to a newly created 1.3 M03 database
  by traversing the 1.2 database. But there maybe a simpler (neo4j
  builtin) way to do this. Any hints about this?
 
  Thanks,
  ---
  balazs
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 
 
 
 
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] String statistics after 1.3.M03

2011-02-27 Thread Tobias Ivarsson
The tool is theoretically safe, and has never caused any problems during
testing.
It does however come with NO GUARANTEES, and running it is on your own risk.

I'm not sure the input from those statistics would influence anything at
this point, but since I haven't looked at any of these statistics of the
string store WITH short strings in place it would at least be fun to look
at. The information to be found in that data is: how many strings
are reasonably short (could have been covered by one of the originally
proposed short string encodings), but were not covered by the actual
implementation.

However, when implementing this optimization I found an embarrassing bug in
my statistics code. Nothing that will harm your data, but rather make the
statistics slightly misleading. Some sort of bonus points to anyone who
finds it. I should perhaps fix that bug if the tool is to be used further.
But on the other hand, not fixing it means that the statistics would show me
how much the implementation misses because of my mistake in coding up the
statistics tool, and I would actually find that interesting.

I don't think I'll have time to investigate any statistics in quite a while.
So if you do decide to run the tool and send data to me, I might not respond
for a few weeks.

Cheers,
Tobias

On Sun, Feb 27, 2011 at 6:44 PM, Massimo Lusetti mluse...@gmail.com wrote:

 Hi neo4j developers,
  I'm repopulating a neo4j db with fresh data made after M03 release,
 so I guess If I could run Tobias statistics tool to gather some
 info... are these interesting to you and safe for data?

 Cheers
 --
 Massimo
 http://meridio.blogspot.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] MMap Error on importing large data

2011-02-27 Thread Tobias Ivarsson
, it seems like keeping mmap settings relatively small
 should
   avoid this problem, although might not lead to best performance? Have I
   understood correctly?
  
   On Windows heap buffers are used by default and auto configuration
   will look how much heap is available. Getting out of memory exceptions
   is an indication that the configuration passed in is using more memory
   than available heap.
  
  
   I am currently using -Xmx2048 on a 4GB ram machine, 32bit java, and the
   settings:
  
  static {
  NORMAL_CONFIG.put( neostore.nodestore.db.mapped_memory, 50M
 );
  NORMAL_CONFIG.put(
   neostore.relationshipstore.db.mapped_memory, 150M );
  NORMAL_CONFIG.put( neostore.propertystore.db.mapped_memory,
  200M );
  NORMAL_CONFIG.put(
   neostore.propertystore.db.strings.mapped_memory, 300M );
  NORMAL_CONFIG.put(
   neostore.propertystore.db.arrays.mapped_memory, 10M );
  NORMAL_CONFIG.put( dump_configuration, false );
  }
  
  
   These settings do not seem to be too high, but if the normal graph
  database
   service will allocate memory outside the heap, and the heap has already
  been
   filled by the batch inserter, perhaps that is where the problem lies?
   Perhaps we do need a way of freeing memory more aggressively after the
  batch
   insertion phase? Get the heap down before allowing the normal API
 access
  to
   the memory?
  
   Regards, Craig
   ___
   Neo4j mailing list
   User@lists.neo4j.org
   https://lists.neo4j.org/mailman/listinfo/user
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] ServerPlugin example GetAll failed to load

2011-02-27 Thread Tobias Ivarsson
Looks like we should augment that exception to include some information
about what the name was that it didn't except :)

-tobias

On Sun, Feb 27, 2011 at 6:49 PM, Andreas Kollegger 
andreas.kolleg...@neotechnology.com wrote:

 Hi again,

 So, it looks like the verifyName() complaint is about the name containing
 characters that aren't URL encoded.

 And, sorry, you asked for the source to ServerPlugin.java. That's here:

 https://github.com/neo4j/server/blob/master/server-api/src/main/java/org/neo4j/server/plugins/ServerPlugin.java

 Cheers,
 Andreas

 On Feb 27, 2011, at 4:27 PM, Brendan Cheng wrote:

  Hi,
 
  I tried to load the GetAll example into ServerPlugin directory of
  1.3.M03 but found:
 
  22:58:28 | 850 [WrapperSimpleAppMain] WARN
  org.neo4j.server.plugins.PluginManager  - Failed to load plugin:
  ServerPlugin[GetAll]
  INFO   | jvm 1| 2011/02/27 22:58:28 |
  java.lang.IllegalArgumentException: Name contains illegal characters
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
  org.neo4j.server.plugins.ServerPlugin.verifyName(ServerPlugin.java:120)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
  org.neo4j.server.plugins.PluginPoint.init(PluginPoint.java:36)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
  org.neo4j.server.plugins.PluginMethod.init(PluginMethod.java:40)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
 
 org.neo4j.server.plugins.PluginPointFactoryImpl.createFrom(PluginPointFactoryImpl.java:98)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
 
 org.neo4j.server.plugins.ServerPlugin.getDefaultExtensionPoints(ServerPlugin.java:175)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
 
 org.neo4j.server.plugins.ServerPlugin.loadServerExtender(ServerPlugin.java:152)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
  org.neo4j.server.plugins.PluginManager.init(PluginManager.java:57)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
  org.neo4j.server.plugins.PluginManager.init(PluginManager.java:45)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
  org.neo4j.server.modules.RESTApiModule.loadPlugins(RESTApiModule.java:75)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
  org.neo4j.server.modules.RESTApiModule.start(RESTApiModule.java:50)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
 
 org.neo4j.server.NeoServerWithEmbeddedWebServer.startModules(NeoServerWithEmbeddedWebServer.java:114)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
 
 org.neo4j.server.NeoServerWithEmbeddedWebServer.start(NeoServerWithEmbeddedWebServer.java:83)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
  org.neo4j.server.BootStrapper.start(BootStrapper.java:62)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
  org.neo4j.server.BootStrapper.main(BootStrapper.java:115)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
  sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
 
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
  java.lang.reflect.Method.invoke(Method.java:597)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
 
 org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:272)
  INFO   | jvm 1| 2011/02/27 22:58:28 | at
  java.lang.Thread.run(Thread.java:662)
 
  I have changed a character from your example.
  Any idea?
  where can I locate your ServerPlugin.java file?
  Cheers,
 
  Brendan
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Two new milestones: Neo4j 1.3.M03 and Spring Data Graph 1.0.0.M3 Released

2011-02-26 Thread Tobias Ivarsson
On Sat, Feb 26, 2011 at 2:05 PM, rick.bullo...@burningskysoftware.com 
rick.bullo...@burningskysoftware.com wrote:

 Also, I assume the new string storage system works with existing neo
 databases?


Yes, as I have explained in my emails to this list before, and in my blog
post describing this feature (linked from the release announcement), this
feature is fully backwards compatible.

-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


[Neo4j] Two new milestones: Neo4j 1.3.M03 and Spring Data Graph 1.0.0.M3 Released

2011-02-25 Thread Tobias Ivarsson
Friends!

Today the Neo4j team released milestone 3 of Neo4j 1.3.

This release features some interesting kernel improvements, such as a better
storage of short strings.
It also improves the Neo4j graph database server. For example by making it
easier to write server plugins.

For more details, please see the release blog post:
http://blog.neo4j.org/2011/02/neo4j-13-m03-with-short-strings.html

As if that was not enough, we then took that fresh new release and included
it in Spring Data Graph. The component for integrating Neo4j with Spring
that we are building together with VMWare/SpringSource. Spring Data Graph
version 1.0.0 was then also released in it's third milestone! This release
contains soo many nice things that the best way to tell you about it is by
including the original release note (forwarded below).

For downloading Neo4j 1.3, milestone 3, please go to:
http://neo4j.org/download
For downloading Spring Data Graph, please see the official release
announcement: http://www.springsource.org/node/3040

I would like to thank everyone who were involved in making this release
possible, especially all of you who helped me with gathering statistics for
better short string support.

Please try out the new features in these releases, and continue to give us
feedback. We really appreciate the mutual support in this community.

Cheers,
Tobias

-- Forwarded message --
From: Thomas Risberg
Date: Sat, Feb 26, 2011 at 12:03 AM
Subject: Spring Data Graph - Neo4j Support 1.0.0.M3 Released


Hi everyone,

We are pleased to announce that a new milestone release (1.0.0.M3) of the
Spring Data Graph 1.0
project with Neo4j support is now available. The new M3 release provides:

* update to Neo4j-1.3.M03, AspectJ 1.6.11.M2
* added explicit support for detachable entities
* added bean validation support for property changes
* added external transaction manager support (XA and one phase commit best
effort)
* added xml namespace configuration
* simplified Spring Java Config configuration class
* added automatic numerical indexing and range queries
* added full featured Neo4jTemplate
* all methods introduced to entities are now generified

and updated reference documentation.

See the official announcement at http://www.springsource.org/node/3040

Feedback welcome!
--
Thomas

-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Traversal Framework question

2011-02-20 Thread Tobias Ivarsson
On Sun, Feb 20, 2011 at 10:48 AM, Alfredas Chmieliauskas 
al.fre...@gmail.com wrote:

 Dear all,

 could somebody point me to more documentation on the new traversal
 framework (besides http://wiki.neo4j.org/content/Traversal_Framework)?
 Also the new Evaluator and how to use it?


That page, along with the examples pages is the best there is:
* http://components.neo4j.org/neo4j-examples/snapshot/traversal.html
* http://wiki.neo4j.org/content/Traversal_HowTo



 If we have a graph described in the pipes Co-Developers example
 (https://github.com/tinkerpop/pipes/wiki/Using-Pipes-to-Traverse-Graphs).
 Would this traversal return the co-developers?

 Traversal.description()
 .depthFirst()
 .relationships(RelationshipTypes.CREATED, Direction.OUTGOING)
 .relationships(RelationshipTypes.CREATED, Direction.INCOMING)
 .traverse(developer).nodes()


No, unfortunately it wouldn't. Relationship type specifications in
TraversalDescriptions are not ordered, what you have written is just a
different spelling of:

Traversal.description()
.depthFirst()
.relationships(RelationshipTypes.CREATED, Direction.BOTH)
.traverse(developer).nodes()

Cheers,
-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Better support for large property data

2011-02-20 Thread Tobias Ivarsson
. The drawback of this would be that memory mapping
would be spread over four files instead of only two.

Keep the comments coming.

Cheers,
Tobias

 -Original Message-
 From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org]
 On
 Behalf Of Tobias Ivarsson
 Sent: Friday, February 18, 2011 9:20 AM
 To: Neo user discussions
 Subject: [Neo4j] Better support for large property data

 Having tackled short strings, I feel up for taking a stab at long strings,
 and large binary data objects.

 I know that Rick Bullotta is really interested in this, and I can imagine
 others wanting to store large properties as well. I would love to get your
 input on the ideas I have, as well as hearing about the ideas you might
 have.

 The way I see it there are two different kinds of large data objects.

 The first one is long strings, or text. Imagine building a blog engine on
 Neo4j, the text body of a blog post is likely going to be around a thousand
 characters. That is a lot of blocks in the DynamicStringStore. But you
 still
 want to support shorter strings (the title of the post for example),
 without
 much overhead, so you don't want to increase the block size for the
 DynamicStringStore. In your code you want to deal with these values as
 String objects though, you don't want a different object type just because
 the string happens to be longer.

 The second one is large binary data objects. Data objects that are too
 large
 to want to have allocated as a String object, or even as a byte[] object.
 You want to manipulate them through some sort of streaming interface. These
 data objects are also so large that you would prefer if their content
 wasn't
 written to the transaction logs, because that would mean that Neo4j needed
 to rotate the log extremely frequently, and since you keep the logical logs
 for HA and backup, it would fill up your disks twice as quickly as it
 needed. Properties like this would, for example, be used for storing images
 that are included in the blog posts.


 For long Strings (the first point), the solution I'm thinking of is to
 replace the stringstore and arraystore with a smallstore and a largestore.
 Both being dynamic block stores as they are today, but with different block
 sizes. Then store both arrays and strings in both of these stores. The type
 of the data stored in the block is stored in the property record for the
 property that references the blocks anyhow, so there isn't a great
 advantage
 of having different block stores for strings and arrays.

 For BLOBs (the second point), we need additions to the API, since you want
 to work with these things in a streaming fashion.
 I am thinking that we use java.nio.channels.ReadableByteChannel for these
 properties. Why ReadableByteChannel you ask? Why not InputStream?
 First reason: InputStream can be converted to ReadableByteChannel, and vice
 versa:

 http://download.oracle.com/javase/6/docs/api/index.html?java/nio/channels/Ch
 annels.htmlhttp://download.oracle.com/javase/6/docs/api/index.html?java/nio/channels/Channels.html
 Second reason: ReadableByteChannel is a really simple interface (only three
 methods) if you want to write your own custom implementation.

 Setting a BLOB property would then look like this:

 ReadableByteChannel myBlob = ...
 node.setProperty(a_blob, myBlob);

 Getting would look like this:

 ReadableByteChannel myBlob =
 (ReadableByteChannel)node.getProperty(a_blob);


 Perhaps we could then, also come up with some nice API for appending to a
 BLOB property:

 ReadableByteChannel moreData = ...
 ReadableByteChannel myBlob =
 (ReadableByteChannel)node.getProperty(a_blob);
 node.setProperty( a_blob, BlobUtils.append(myBlob, moreData) );


-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Online backup works in full but not incremental

2011-02-20 Thread Tobias Ivarsson
On Sun, Feb 20, 2011 at 6:11 PM, Brendan ccp...@gmail.com wrote:

 Hi,

 After I install the neo4j on ubuntu server I'm able to backup the full
 database, even repeatedly but it crashed on incremental.


Could you please provide a stacktrace, and other kinds of error output from
this crash.

Thank you,
-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Traversal Framework question

2011-02-20 Thread Tobias Ivarsson
On Sun, Feb 20, 2011 at 7:04 PM, Marko Rodriguez okramma...@gmail.comwrote:

 Hi,

  Traversal.description()
  .depthFirst()
  .relationships(RelationshipTypes.CREATED, Direction.BOTH)
  .traverse(developer).nodes()

 To be clear, a co-creator is someone is who has created the same things as
 you and who is not you. Thus, you need to go outgoing CREATED, then incoming
 CREATED, then you need to make sure that the vertex you land at is not the
 one you left from -- thus, you need to filter the originating vertex.


The easiest way to do that in this case is by adding:

.uniqueness(Uniqueness.NODE_PATH)

Cheers,
-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Traversal Framework question

2011-02-20 Thread Tobias Ivarsson
Marko, I don't understand what you are trying to say.

Is the question about finding, given a person node, the co-creators for that
person on any project? Or is it about reasoning in the abstract co-creator
graph. I was under the impression that finding co-creators was the only
thing we were interested in, in that case path-uniqueness will do the trick.

What node-path-uniqueness guarantees is that within each path the same node
occurs at most once. In a path from a creator to her co-creator that would
mean that the creator cannot be her own co-creator. If the starting person
has created both x and y, the path through x and the path through y will be
different paths, which means that the path uniqueness will allow the same
persons to be co-creator on both projects.

Path uniqueness would get you into problem if person A created person B and
person B (who was created) was also a creator of herself, then person B
would not be discovered as a co-creator. I cannot imagine a case where that
would be sensible. It is also impossible to store in Neo4j, since
self-relationships are not allowed.

You can do something like .filter(from two steps ago) by looking two steps
back in the path of what you've traversed so far in the evaluator.

I fear that if we continue further on the path where this discussion is
currently headed, it will become too theoretical to be of use to anyone.
Let's keep such discussions isolated.

Cheers,
Tobias

On Sun, Feb 20, 2011 at 7:14 PM, Marko Rodriguez okramma...@gmail.comwrote:

 Hi,

 Just to be picky:

  The easiest way to do that in this case is by adding:
  .uniqueness(Uniqueness.NODE_PATH)

 A co-creator's co-creator can be you. Thus, marko's co-creator's co-creator
 is marko (amongst other people). In this case, unique on a path would not
 fail, no? Can you do something like .filter(from two steps ago)?

 Thanks,
 Marko.




 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Neovigator Weekend Project

2011-02-20 Thread Tobias Ivarsson
Nice stuff!

On Mon, Feb 21, 2011 at 6:18 AM, Max De Marzi Jr. maxdema...@gmail.comwrote:

 Guys,

 So I ran into the Ask Ken project ( http://askken.heroku.com/ ) by
 Michael Aufreiter yesterdat, and though it was pretty awesome... so I
 ported it to using Neo4j.

 Check it out: http://neovigator.heroku.com/

 On github at https://github.com/maxdemarzi/neovigator

 Regards,
 Max
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Help us make Neo4j better at handling YOUR data

2011-02-18 Thread Tobias Ivarsson
Thank you everyone who submitted statistics, both in this thread and to me
directly!

As of last night, this feature is committed in Neo4j trunk, and should make
it into the next milestone release, at which point the sum of you guys
string stores should be possible to store in 65% less space.

I've written up a blog post about how the implementation works:
http://journal.thobe.org/2011/02/better-support-for-short-strings-in.html

Thank you so much for all your help!
-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Slow down on insertion as db grow

2011-02-18 Thread Tobias Ivarsson
On Thu, Feb 17, 2011 at 3:32 PM, Massimo Lusetti mluse...@gmail.com wrote:

 I'm trying to populate the DB with Nodes with only one String
 property, an IPv4 address. This is just a test case for my typical use
 case and it shows that when the process starts the first 440744 nodes
 are going in at a speed of 1.05ms each Node but this speed drops to
 8.75ms after having inserted 12545155 nodes.
 The final DB size is: 2.9G since i tweaked the sintrg_block_size at
 graphdb creation time to 60bytes instead of 120...


I recently (yesterday) committed a new feature for Neo4j that will store
these kinds of short strings, making Neo4j store them without having to
involve the DynamicStringStore at all. You should see a substantial speedup
from using that. IPv4 addresses will always be storeable as a short string,
so if all you store as properties are IPv4 addresses the DynamicStringStore
wouldn't be used at all in your use case.

I've written a bit about how it works on my blog:
http://journal.thobe.org/2011/02/better-support-for-short-strings-in.html

The feature is available in the latest snapshot build:
http://neo4j.org/get?file=neo4j-1.3-SNAPSHOT-unix.tar.gz
And will be released in the next milestone.


WRT the slowdown you've seen, I'll have to investigate that further.

Cheers,
-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Help us make Neo4j better at handling YOUR data

2011-02-18 Thread Tobias Ivarsson
Yes, I'm thinking of making that my lab project for this weekend.

I'm starting a separate thread for discussing that, title: Better support
for large property data. I'd love your input Rick.

-tobias

On Fri, Feb 18, 2011 at 2:43 PM, Rick Bullotta 
rick.bullo...@burningskysoftware.com wrote:

 Awesome!  Now wanna tackle the other end of the spectrum, really big
 strings
 and byte arrays? ;-)



 -Original Message-
 From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org]
 On
 Behalf Of Tobias Ivarsson
 Sent: Friday, February 18, 2011 8:29 AM
 To: Neo user discussions
 Subject: Re: [Neo4j] Help us make Neo4j better at handling YOUR data

 Thank you everyone who submitted statistics, both in this thread and to me
 directly!

 As of last night, this feature is committed in Neo4j trunk, and should make
 it into the next milestone release, at which point the sum of you guys
 string stores should be possible to store in 65% less space.

 I've written up a blog post about how the implementation works:
 http://journal.thobe.org/2011/02/better-support-for-short-strings-in.html

 Thank you so much for all your help!
 --
 Tobias Ivarsson tobias.ivars...@neotechnology.com
 Hacker, Neo Technology
 www.neotechnology.com
 Cellphone: +46 706 534857
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


[Neo4j] Better support for large property data

2011-02-18 Thread Tobias Ivarsson
Having tackled short strings, I feel up for taking a stab at long strings,
and large binary data objects.

I know that Rick Bullotta is really interested in this, and I can imagine
others wanting to store large properties as well. I would love to get your
input on the ideas I have, as well as hearing about the ideas you might
have.

The way I see it there are two different kinds of large data objects.

The first one is long strings, or text. Imagine building a blog engine on
Neo4j, the text body of a blog post is likely going to be around a thousand
characters. That is a lot of blocks in the DynamicStringStore. But you still
want to support shorter strings (the title of the post for example), without
much overhead, so you don't want to increase the block size for the
DynamicStringStore. In your code you want to deal with these values as
String objects though, you don't want a different object type just because
the string happens to be longer.

The second one is large binary data objects. Data objects that are too large
to want to have allocated as a String object, or even as a byte[] object.
You want to manipulate them through some sort of streaming interface. These
data objects are also so large that you would prefer if their content wasn't
written to the transaction logs, because that would mean that Neo4j needed
to rotate the log extremely frequently, and since you keep the logical logs
for HA and backup, it would fill up your disks twice as quickly as it
needed. Properties like this would, for example, be used for storing images
that are included in the blog posts.


For long Strings (the first point), the solution I'm thinking of is to
replace the stringstore and arraystore with a smallstore and a largestore.
Both being dynamic block stores as they are today, but with different block
sizes. Then store both arrays and strings in both of these stores. The type
of the data stored in the block is stored in the property record for the
property that references the blocks anyhow, so there isn't a great advantage
of having different block stores for strings and arrays.

For BLOBs (the second point), we need additions to the API, since you want
to work with these things in a streaming fashion.
I am thinking that we use java.nio.channels.ReadableByteChannel for these
properties. Why ReadableByteChannel you ask? Why not InputStream?
First reason: InputStream can be converted to ReadableByteChannel, and vice
versa:
http://download.oracle.com/javase/6/docs/api/index.html?java/nio/channels/Channels.html
Second reason: ReadableByteChannel is a really simple interface (only three
methods) if you want to write your own custom implementation.

Setting a BLOB property would then look like this:

ReadableByteChannel myBlob = ...
node.setProperty(a_blob, myBlob);

Getting would look like this:

ReadableByteChannel myBlob =
(ReadableByteChannel)node.getProperty(a_blob);


Perhaps we could then, also come up with some nice API for appending to a
BLOB property:

ReadableByteChannel moreData = ...
ReadableByteChannel myBlob =
(ReadableByteChannel)node.getProperty(a_blob);
node.setProperty( a_blob, BlobUtils.append(myBlob, moreData) );


Comment please.
-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Help with exception using BatchInserter

2011-02-15 Thread Tobias Ivarsson
I had a second look at this, and realized that those memory mapping settings
aren't used from the batch inserter. Everything is allocated in the Java
Heap. 63G out of 64G RAM is still a bit much, but not as big of a problem as
I first thought.

This also means that there could be another problem behind the exception you
receive. There should have been a causing exception for the one you listed.
Could you please provide a full stack trace?

Cheers,
Tobias

On Thu, Feb 10, 2011 at 10:42 PM, Pablo Pareja ppar...@era7.com wrote:

 Hi all,
 I keep getting this exception and don't know what the reason could be:

 Unable to write record[244986398] @[32583190934]

 org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112)

 org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124)

 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251)

 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224)

 org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614)

 org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101)

 org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611)
 org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115)

 org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611)

 org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351)

 I'm using a 65G RAM machine and the java process is launched with -Xmx63G
 the conf values I pass to the batch inserter are the following:

 neostore.nodestore.db.mapped_memory=10G
 neostore.relationshipstore.db.mapped_memory=10G
 neostore.propertystore.db.mapped_memory=10G
 neostore.propertystore.db.strings.mapped_memory=10G
 neostore.propertystore.db.arrays.mapped_memory=0M

 Besides, there were still 50G free space in the disk when this exception
 was
 thrown

 I'd really appreciate any help,
 Cheers

 --
 Pablo Pareja Tobes
 LinkedInhttp://www.linkedin.com/in/pabloparejatobes
 Twitter   http://www.twitter.com/pablopareja

 http://www.ohnosequences.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Help with exception using BatchInserter

2011-02-15 Thread Tobias Ivarsson
There should have been a cause at the end of that trace, looking something
like:

caused by: java.io.IOException ...

That is what I'm interested in.

-tobias

On Tue, Feb 15, 2011 at 10:12 AM, Pablo Pareja ppar...@era7.com wrote:

 I already put the stack trace in the first message:

 Unable to write record[244986398] @[32583190934]

 org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112)

 org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124)

 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251)

 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224)

 org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614)

 org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101)

 org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611)
 org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115)

 org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611)

 org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351)

 the missing lines are just from my own classes ending in a
 BatchInserter.shutdown() method.

 Cheers,

 Pablo

 On Tue, Feb 15, 2011 at 9:06 AM, Tobias Ivarsson 
 tobias.ivars...@neotechnology.com wrote:

  I had a second look at this, and realized that those memory mapping
  settings
  aren't used from the batch inserter. Everything is allocated in the Java
  Heap. 63G out of 64G RAM is still a bit much, but not as big of a problem
  as
  I first thought.
 
  This also means that there could be another problem behind the exception
  you
  receive. There should have been a causing exception for the one you
 listed.
  Could you please provide a full stack trace?
 
  Cheers,
  Tobias
 
  On Thu, Feb 10, 2011 at 10:42 PM, Pablo Pareja ppar...@era7.com wrote:
 
   Hi all,
   I keep getting this exception and don't know what the reason could be:
  
   Unable to write record[244986398] @[32583190934]
  
  
 
 org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112)
  
  
 
 org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124)
  
  
 
 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251)
  
  
 
 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224)
  
  
 
 org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614)
  
  
 
 org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101)
  
  
 
 org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611)
  
 
 org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115)
  
  
 
 org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611)
  
  
 
 org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351)
  
   I'm using a 65G RAM machine and the java process is launched with
 -Xmx63G
   the conf values I pass to the batch inserter are the following:
  
   neostore.nodestore.db.mapped_memory=10G
   neostore.relationshipstore.db.mapped_memory=10G
   neostore.propertystore.db.mapped_memory=10G
   neostore.propertystore.db.strings.mapped_memory=10G
   neostore.propertystore.db.arrays.mapped_memory=0M
  
   Besides, there were still 50G free space in the disk when this
 exception
   was
   thrown
  
   I'd really appreciate any help,
   Cheers
  
   --
   Pablo Pareja Tobes
   LinkedInhttp://www.linkedin.com/in/pabloparejatobes
   Twitter   http://www.twitter.com/pablopareja
  
   http://www.ohnosequences.com
   ___
   Neo4j mailing list
   User@lists.neo4j.org
   https://lists.neo4j.org/mailman/listinfo/user
  
 
 
 
  --
  Tobias Ivarsson tobias.ivars...@neotechnology.com
  Hacker, Neo Technology
  www.neotechnology.com
  Cellphone: +46 706 534857
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 



 --
 Pablo Pareja Tobes
 LinkedInhttp://www.linkedin.com/in/pabloparejatobes
 Twitter   http://www.twitter.com/pablopareja

 http://www.ohnosequences.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Help with exception using BatchInserter

2011-02-14 Thread Tobias Ivarsson
Your problem is likely related to the fact that 10G+10G+10G+10G + 63G  64G.

Memory mapping uses memory outside of the Java heap. All memory you use,
memory mapping plus Java heap must fit in the available RAM.

Cheers,
Tobias

On Thu, Feb 10, 2011 at 10:42 PM, Pablo Pareja ppar...@era7.com wrote:

 Hi all,
 I keep getting this exception and don't know what the reason could be:

 Unable to write record[244986398] @[32583190934]

 org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112)

 org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124)

 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251)

 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224)

 org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614)

 org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101)

 org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611)
 org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115)

 org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611)

 org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351)

 I'm using a 65G RAM machine and the java process is launched with -Xmx63G
 the conf values I pass to the batch inserter are the following:

 neostore.nodestore.db.mapped_memory=10G
 neostore.relationshipstore.db.mapped_memory=10G
 neostore.propertystore.db.mapped_memory=10G
 neostore.propertystore.db.strings.mapped_memory=10G
 neostore.propertystore.db.arrays.mapped_memory=0M

 Besides, there were still 50G free space in the disk when this exception
 was
 thrown

 I'd really appreciate any help,
 Cheers

 --
 Pablo Pareja Tobes
 LinkedInhttp://www.linkedin.com/in/pabloparejatobes
 Twitter   http://www.twitter.com/pablopareja

 http://www.ohnosequences.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Does the index commit with the transaction?

2011-02-14 Thread Tobias Ivarsson
That is correct. The lucene indexes in Neo4j are tied to the same
transaction life cycle.

On Sun, Feb 13, 2011 at 6:18 PM, Massimo Lusetti mluse...@gmail.com wrote:

 Hi all,
  Does the inner Lucene index commit toghether with the Transaction
 success()/finish() cycle?

 I mean if I start a Transaction and do like
 http://wiki.neo4j.org/content/Transactions#Big_transactions I suppose
 I'm guarantee the Lucene index is synched to disk as soon as the
 Transaction finish(), right?

 Cheers
 --
 Massimo
 http://meridio.blogspot.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Does BatchInserter auto-commit before shutting down?

2011-02-14 Thread Tobias Ivarsson
The batch inserter will write data to disk as needed. The memory mapped
regions are used to speed up writes, so that the API level writes can be
done to the memory mapped regions which can then be written in large
sequential chunks to disk.

You do not have to shut down the batch inserter for this to happen, it is
taken care of automatically.

Cheers,
Tobias

On Mon, Feb 14, 2011 at 10:53 AM, Pablo Pareja ppar...@era7.com wrote:

 Hi,

 I was wondering whether once the memory-mapping configuration for
 BatchInserter is full *(for example*
 *in the case of  neostore.relationshipstore.db.mapped_memory) *the class
 would auto-commit these changes
 to disk or it would just throw an exception.
 In the latter case, what would the options be?

 In my case I have to insert a huge amount of data that cannot be allocated
 in RAM.
 First I thought the BatchInserter would deal with this kind of situation,
 however I keep getting exceptions whenever the
 program reaches the point where *(I guess) *memory mapping configuration is
 not enough.

 Since I cannot use more RAM than what I have, I thought, ok so I will
 shutdown both inserter and index service and
 create them again each time the program have already inserted a decent
 amount of data.
 Then, don't know why but after a couple of tests, at some point the program
 always halts throwing a really weird exception
 saying something about a index file that exists but cannot be accessed in
 the merge process ?!?

 I guess I'd be doing something wrong but cannot figure out what it is.
 Thanks in advance

 --
 Pablo Pareja Tobes
 LinkedInhttp://www.linkedin.com/in/pabloparejatobes
 Twitter   http://www.twitter.com/pablopareja

 http://www.ohnosequences.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Does BatchInserter auto-commit before shutting down?

2011-02-14 Thread Tobias Ivarsson
BatchInserter.shutdown() is synchronous, it has to be, otherwise it wouldn't
be safe at all.
After shutdown() has returned it guarantees that all store files have been
written, and are in a consistent state.

-t

On Mon, Feb 14, 2011 at 4:01 PM, Pablo Pareja ppar...@era7.com wrote:

 Great, thanks for the information.
 Just out of curiosity then, are calls to BatchInserter.shutdown() sync or
 asynchronous ?
 I mean, does the program flow stops in that call till every operation the
 shutdown implies has been
 performed?
 Cheers,

 Pablo

 On Mon, Feb 14, 2011 at 3:30 PM, Tobias Ivarsson 
 tobias.ivars...@neotechnology.com wrote:

  The batch inserter will write data to disk as needed. The memory mapped
  regions are used to speed up writes, so that the API level writes can be
  done to the memory mapped regions which can then be written in large
  sequential chunks to disk.
 
  You do not have to shut down the batch inserter for this to happen, it is
  taken care of automatically.
 
  Cheers,
  Tobias
 
  On Mon, Feb 14, 2011 at 10:53 AM, Pablo Pareja ppar...@era7.com wrote:
 
   Hi,
  
   I was wondering whether once the memory-mapping configuration for
   BatchInserter is full *(for example*
   *in the case of  neostore.relationshipstore.db.mapped_memory) *the
 class
   would auto-commit these changes
   to disk or it would just throw an exception.
   In the latter case, what would the options be?
  
   In my case I have to insert a huge amount of data that cannot be
  allocated
   in RAM.
   First I thought the BatchInserter would deal with this kind of
 situation,
   however I keep getting exceptions whenever the
   program reaches the point where *(I guess) *memory mapping
 configuration
  is
   not enough.
  
   Since I cannot use more RAM than what I have, I thought, ok so I will
   shutdown both inserter and index service and
   create them again each time the program have already inserted a decent
   amount of data.
   Then, don't know why but after a couple of tests, at some point the
  program
   always halts throwing a really weird exception
   saying something about a index file that exists but cannot be accessed
 in
   the merge process ?!?
  
   I guess I'd be doing something wrong but cannot figure out what it is.
   Thanks in advance
  
   --
   Pablo Pareja Tobes
   LinkedInhttp://www.linkedin.com/in/pabloparejatobes
   Twitter   http://www.twitter.com/pablopareja
  
   http://www.ohnosequences.com
   ___
   Neo4j mailing list
   User@lists.neo4j.org
   https://lists.neo4j.org/mailman/listinfo/user
  
 
 
 
  --
  Tobias Ivarsson tobias.ivars...@neotechnology.com
  Hacker, Neo Technology
  www.neotechnology.com
  Cellphone: +46 706 534857
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 



 --
 Pablo Pareja Tobes
 LinkedInhttp://www.linkedin.com/in/pabloparejatobes
 Twitter   http://www.twitter.com/pablopareja

 http://www.ohnosequences.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Help us make Neo4j better at handling YOUR data

2011-02-10 Thread Tobias Ivarsson
This tool does not gather statistics for string arrays, since they are not
affected by the proposed patch.

The tool still runs, but if there are no String properties, output will be
pretty boring

Cheers,
Tobias

On Wed, Feb 9, 2011 at 7:04 PM, Massimo Lusetti mluse...@gmail.com wrote:

 On Fri, Feb 4, 2011 at 5:13 PM, Tobias Ivarsson
 tobias.ivars...@neotechnology.com wrote:
  Friends,
 
  Please read this, your involvement will make Neo4j more efficient!

 Does this run with a db with only String[] data?!

 Cheers
 --
 Massimo
 http://meridio.blogspot.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Solve a problem using neo4j

2011-02-08 Thread Tobias Ivarsson
Hi John,

The High Availability implementation of Neo4j behaves in the same way.

Reads in HA are done from the local store.

An interesting side effect of this is that while all instances in a HA
cluster will store the same data, they can each have a different dataset in
cache. This can, with some consistent request routing, be used for what we
call cache sharding.

-tobias

On Tue, Feb 8, 2011 at 10:10 AM, yehohan...@googlemail.com 
yehohan...@gmail.com wrote:

 Hello Jim,
 To add to the previous question about lazy loading, how does highavailable
 implementation of neo4j behave?

 John

 Sent from my Nokia phone
 -Original Message-
 From: Jim Webber
 Sent:  08/02/2011 09:16:36
 Subject:  Re: [Neo4j] Solve a problem using neo4j

 Hi Ori,

  1. About the iterator - if I get a large amount of results (millions)  it
  cannot be held in memory. Does the iterator work on in-memory collection,
 or
  there is something like JDBC fetch size?

 For the embedded case, iterators are lazy so you won't hurt yourself. For
 the REST API results of traversers are built up in memory so you can hurt
 yourself.

  2. Is there an example for what I'm looking for or can someone give
 pseudo
  code example for #3?

 That's pretty domain specific. But I found that looking at the Social
 Network example gave me enough understanding of the simple traverser API to
 get going:

 http://blog.neo4j.org/2009/09/social-networks-in-database-using-graph.html
 http://components.neo4j.org/neo4j-examples/1.2-SNAPSHOT/social-network.html

 Jim


 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Calculating shortest paths in a large graph

2011-02-08 Thread Tobias Ivarsson
.getNodesInCheapestPath(generated)
  
  
  
   Regards
   From: sxk1...@hotmail.com
   To: user@lists.neo4j.org
   Date: Sun, 30 Jan 2011 07:41:00 -0800
   Subject: Re: [Neo4j] Calculating shortest paths in a large
   graph
  
  
   Hello Tobias,Thanks very much for your help, to be clear I want
   to
   find
   the path that has the least cost, in this case length is the
   determinator
   for doing this.  The total number of nodes are roughly around a
 100
   but
   not
   much more.  I am attaching my heap dump,my code for parsing the
   google
   earth
   data as well as the actual xml file I use to parse and load the
   data,
   managing the Graph and the relationships.
   My project does the following:1) Create a physical graph based
   on
   google earth with a set of nodes and paths2) Load the contents of
   the
   google
   earth file into neo4j using regular expressions to parse the xml3)
   Run
   the
   dijkstra algorithm on this graph
  
  
   Your help is much appreciated, let me know if there's anything
   else
   I
   can provide.Regards
  
   From: tobias.ivars...@neotechnology.com
   Date: Sun, 30 Jan 2011 16:22:42 +0100
   To: user@lists.neo4j.org
   Subject: Re: [Neo4j] Calculating shortest paths in a large
   graph
  
   Hi Saikat,
  
   This was a strange one.
  
   Are you sure you only have 100 nodes? That is really tiny, and
   your
   title
   said large graph. How many relationships?
  
   I really cant see a way this could run into heap space issues,
   what
   does
   your heap configuration look like?
  
   1) Batch insertion will not help you, since you aren't
   inserting,
   you
   are
   reading from the graph.
   2) Granted, Dijkstra isn't the best algorithm for finding the
   cheapest
   path
   (that's what it finds, not shortest path), but with a small
   graph
   it
   should
   still terminate quickly.
   3) Unless you are running this on a machine with only 8MB of
   RAM
   (or
   something equally silly), hardware should not be your problem.
  
   If the specifications you've specified are in fact correct,
   and
   your
   graph
   only contains 100 nodes, then could you please provide me with
   a
   heap
   dump
   so that I can analyze where heap is being wasted, because OOM
   on
   such
   a
   small graph would be a bug.
  
   To get a heap dump when the program throws OOME, provide the
   following
   startup parameter to the JVM:
   -XX:+HeapDumpOnOutOfMemoryError
   You can send the resulting .hprof-file to me directly, the
   mailing
   list is a
   bit restrictive about attachments.
   If the graph is larger, and the 100 nodes part was a typo,
   then
   switching
   algorithm might be a good solution, Dijkstra is not optimal
   for
   large
   graphs.
  
   I'd love to help, but I need some more details, because this
   seems
   strange
   to me.
   Cheers,
   Tobias
  
   On Sun, Jan 30, 2011 at 3:19 PM, Saikat Kanjilal 
   sxk1...@hotmail.com
   wrote:
  
  
   Hi Folks,I'm working on a little route planning spring based
   neo4j
   service
   where I initially  load up all my data into neo4j and have
   about
   a
   100
   nodes, however it seems that I am running into heap space
   issues
   when
   running the Dijkstra Algorithm for any traversals that are
   relatively
   far
   apart (i.e. opposite ends of the graph).  I have tried to
   increase
   the heap
   space but that didn't seem to make a difference.   Note that
   I am
   not
   currently using batch insertion and am using neo4j's
   transaction
   manager
   with spring with the @Transactional annotation to perform all
   transactions.
   I was wondering what the areas are that I can look at to get
   past
   this.
   The things that are coming to mind include:
   1) Adding batch insertion2) Trying the other algorithms like
   AStar3)
   Running this on more powerful hardware
  
   Number 3 seems unlikely based on all I've read on neo4j and
   its
   performance
   characteristics.  I can attach my code if needed but being a
   newbie
   to neo4j
   I wanted to understand what I'm missing conceptually that
   could
   be
   causing
   these issues.
   Best Regards
  
  
  
   ___
   Neo4j mailing list
   User@lists.neo4j.org
   https://lists.neo4j.org/mailman/listinfo/user
  
  
  
  
   --
   Tobias Ivarsson tobias.ivars...@neotechnology.com
   Hacker, Neo Technology
   www.neotechnology.com
   Cellphone: +46 706 534857
   ___
   Neo4j mailing list
   User@lists.neo4j.org
   https://lists.neo4j.org/mailman/listinfo/user
  
  
   ___
   Neo4j mailing list
   User@lists.neo4j.org
   https://lists.neo4j.org/mailman/listinfo/user
   ___
   Neo4j mailing list
   User@lists.neo4j.org
   https://lists.neo4j.org/mailman/listinfo/user
  
   ___
   Neo4j mailing list
   User@lists.neo4j.org

Re: [Neo4j] Help us make Neo4j better at handling YOUR data

2011-02-05 Thread Tobias Ivarsson
Damn. That one place assumes that you don't have any empty strings.

I've uploaded a patched version. Same location:
https://github.com/downloads/thobe/neo4j-admin-store/stringstat.jar

-tobias

On Fri, Feb 4, 2011 at 6:52 PM, Rick Bullotta 
rick.bullo...@burningskysoftware.com wrote:

 Same here.

 -Original Message-
 From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org]
 On
 Behalf Of Axel Morgner
 Sent: Friday, February 04, 2011 12:29 PM
 To: user@lists.neo4j.org
 Subject: Re: [Neo4j] Help us make Neo4j better at handling YOUR data

 Hi Tobias,

 just ran the utility, but got an exception:

 Computing character frequencies for 205895 string records
  30%
 ... 40%
 ... 50%
 ... 60%
 ... 70%
 ... 80%
 ... 90%
 ...100%
 Matching potential encodings for 205895 string records
 ...Exception in thread main java.lang.StringIndexOutOfBoundsException:
 String index out of range: 0
 at java.lang.String.charAt(String.java:694)
 at org.neo4j.admin.tool.stringstat.Numerical.matches(Numerical.java:30)
 at

 org.neo4j.admin.tool.stringstat.TryAssumptions.process(TryAssumptions.java:4
 6)
 at org.neo4j.admin.tool.stringstat.Main.main(Main.java:55)

 Greetings

 Axel

  I have written a small utility that analyzes the string properties stored
 by
  Neo4j and computes some statistics about them.
  If I could get as many of you to run this tool on your stores and send
 those
  statistics to me as possible, that would be great.
 
  This tool is available for download here:
  https://github.com/downloads/thobe/neo4j-admin-store/stringstat.jar
 
  To run it, all you need to do is:
  java -jar stringstat.jar /path/to/your/neo4j/store/dir

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user

 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Choosing HTTP method for server plugins (REST API)

2011-02-04 Thread Tobias Ivarsson
I think it makes sense to expose the invocation of a plugin by GET in some
(many/most?) cases, but I don't think it makes sense to expose this to the
plugin author.
REST is not going to be the only transport layer of the server for ever, and
other transport layers don't make the same distinction between
SET/GET/POST/DELETE.

I'm happy to have a discussion about this though :-)

Cheers,
Tobias

On Fri, Feb 4, 2011 at 4:01 PM, Maximilian Schulz nam...@gmail.com wrote:

 I am wondering if it is possible to set the http method for server plugins
 which extend the REST API. The example code only exposes methods which
 respond to POST requests. But I think it would make much more sense to use
 GET in many of my (and the example) cases.

 Max
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


[Neo4j] Help us make Neo4j better at handling YOUR data

2011-02-04 Thread Tobias Ivarsson
:632  0:316
= 6 bit frequencies =
  n: 874452  a: 802639  e: 790998  i: 597945
   : 524600  r: 496695  o: 462202  s: 457249
  l: 429857  d: 327564  t: 310925  .: 217419
  m: 207459  @: 203896  k: 190190  g: 180166
  u: 169502  v: 158119  S: 148109  h: 144968
  M: 118409  H: 101946  b:  93958  c:  85548
  A:  84580  E:  77119  L:  75604  y:  67204
   j:  66711  K:  63279  T:  61919  p:  60935
  f:  60589  B:  51351  J:  48471  ?:  44154
  N:  39113  O:  35251  W:  32999  F:  32734
  G:  30821  R:  30341  z:  27002  V:  25837
  ?:  25434  D:  23706  C:  22801  I:  21277
  ?:  19167  ?:  17891  ?:  14752  P:  13294
  x:  11184  2:  11011  w:   9972  U:   9556
  Y:   7486  -:   4537  Z:   4524  ?:   4496
  1:   4490  4:   3119  q:   1756  0:   1379
   6:   1379  9:   1379  8:   1379  ,:609
  ?:342  ?:294
   9107508 strings with category bitmask  0b0
   2947442 strings with category bitmask0b111
 80361 strings with category bitmask   0b1000
  8251 strings with category bitmask  0b1
 16971 strings with category bitmask  0b10001
 26235 strings with category bitmask  0b100010001
 20632 strings with category bitmask  0b100010011
 41383 strings with category bitmask  0b101010011
 86358 strings with category bitmask  0b10111
   508 strings with category bitmask  0b11000
 32973 strings with category bitmask  0b11001
 26579 strings with category bitmask  0b110010001
= Category index =
 0. NineSevenBitAscii
 1. LowerCaseHexadecimal
 2. UpperCaseHexadecimal
 3. PunctuatedNumerical
 4. AlphaNumericalName
 5. Numerical
 6. FrequencyBased:4bit
 7. FrequencyBased:5bit
 8. FrequencyBased:6bit
/STRING STORE STATISTICS


The information I want you to send back to me is the data between
the STRING STORE STATISTICS and /STRING STORE STATISTICS lines.

May the source be with you,
-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Help us make Neo4j better at handling YOUR data

2011-02-04 Thread Tobias Ivarsson
I actually got an idea on what better support for large binary data would
look like just 5 minutes ago.
I'll try to write a prototype this weekend.

I've been thinking about how to support large binary data for a few months,
as one of my background trails of thought, but I just now got the epiphany
on how it could be exposed to the user.

Cheers,
Tobias

On Fri, Feb 4, 2011 at 5:29 PM, Rick Bullotta 
rick.bullo...@burningskysoftware.com wrote:

 Will do, Tobias.

 Also, I'd love to see a project start soon to help Neo4J become better at
 storing *large* strings/byte arrays, also.  I think with some creativity,
 we
 could avoid the need for the entire large string/blob to be stored in the
 transaction logs, which to me, is one of the big challenges.

 In any case, I'll try to get you statistics soon.

 Rick


-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] neo4j 1.1 Download? -- for Use with Current Python Bindings

2011-02-03 Thread Tobias Ivarsson
Server did not exist for Neo4j 1.1, it was first released with 1.2.

Cheers,
Tobias

On Thu, Feb 3, 2011 at 12:57 AM, James Thornton ja...@jamesthornton.comwrote:

 Thank you. Is the the neo4j 1.1 server download still available?

 On Mon, Jan 31, 2011 at 8:59 PM, James Thornton ja...@jamesthornton.com
 wrote:
  Since the current python bindings only work with neo4j 1.1 (as per
  this bug report https://trac.neo4j.org/ticket/302 and
  http://wiki.neo4j.org/content/Getting_Started_With_Python), where can
  you download neo4j 1.1 in the meantime until the python bindings are
  updated?
 
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Visualization options for graphs

2011-02-03 Thread Tobias Ivarsson
It's a wiki, why didn't you just do it yourself?

This is actually a general question to everyone on the list:
What is the main hurdle preventing you from contributing to the wiki?

Cheers,
Tobias

On Thu, Feb 3, 2011 at 11:04 AM, Luanne Misquitta lmisqui...@saba.comwrote:

 Hi,



 Would be good to include Arbor.js (http://arborjs.org/) to the
 JavaScript options listed in the Neo4J wiki
 (http://wiki.neo4j.org/content/Visualization_options_for_graphs)

 Have just started playing with it and it renders very nice graphs indeed
 (HTML5 + JavaScript).



 Regards

 Luanne M.
 Tech Lead

 twitter / @luannem http://twitter.com/luannem
 linkedin / http://in.linkedin.com/in/luannemisquitta
 http://in.linkedin.com/in/luannemisquitta
 skype / luanne.misquitta
 blog / http://thought-bytes.blogspot.com/
 http://thought-bytes.blogspot.com/
 Saba. Power Up Your People.



 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] InvalidRecordException during traversal after unclean shutdown

2011-02-03 Thread Tobias Ivarsson
I've been able to manually put the store into a consistent state. I did this
by simply skipping the references to Relationship[2055394] that had somehow
gotten into the relationship chain of Node[1]. I haven't, however, been able
to reproduce or find the cause of this.

It would have been interesting to look at the logical log Neo4j created
while working on your store. Could you in the future configure Neo4j with:
keep_logical_logs=true
That way all the logical logs will be kept and I could use that to analyze
the problem, should this happen again, and you could use it to recreate your
store from scratch to a consistent state.

I'll continue my investigation a bit more, but I'm not sure I can get
anywhere further since the faulty record has been removed.

For my continued investigation I have a few questions:

Does your application ever remove any relationships during normal operation?
Or was the removal of the invalid relationship record the only that ever
occurred?
It looks like at least one other, Relationship[2055617], has been removed at
some point.

If this ever happens again, send the faulty store to me immediately, don't
try to fix it on your own. I'd love to see the actual corrupted data. As it
looks at the moment it is tempting to blame vmware (not going to do that
without evidence though), I don't know how well they propagate filesystem
flush() operations to the host file system, I'll have to investigate that.

Was the file system in the vmware vm an actual partition on disk, or was it
a virtual file system stored in an image file on the host?

The non-clean shutdown that caused this, was that a JVM crash, a vmware vm
crash, or a host machine crash?

Cheers,
Tobias

On Tue, Feb 1, 2011 at 7:12 PM, Lars Hvile lars.hv...@fovea.no wrote:

 -- directly after successful recovery
 org.neo4j.kernel.impl.nioneo.store.InvalidRecordException: Node[1] is
 neither firstNode[2055881] nor secondNode[2055876] for Relationship[2055394]


 -- after manual deletion of Relationship[2055394], in a desperate
 recovery-attempt =)


 Unfortunately I don't have a copy of the database at it's original corrupt
 state, I do however have a copy of it after I deleted the the relationship..
 788MiB zipped


It really is a shame that you removed that relationship. It would have been
super useful to look at what that record looked like.
Although I think that record was fine, more likely it was the reference to
it that was faulty.

-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Lucene index commit rate and NoSuchElementException

2011-02-02 Thread Tobias Ivarsson
More threads != faster

You are doing I/O bound work. More then two threads is most likely just
going to add overhead and make things slower!

Also, I'm wondering, what does crunch mean in this context? Is it the
write operations we have been talking about, or is it some other operation?
I'm thinking that if it takes 45 minutes to insert a days worth of data,
that is fast enough to do in real time. If Neo4j allows you to *process*
that data faster and in new ways, then that would be gain enough, and
probably warrant slightly longer insert times.

-tobias

On Wed, Feb 2, 2011 at 12:17 PM, Massimo Lusetti mluse...@gmail.com wrote:

 On Tue, Feb 1, 2011 at 10:19 PM, Tobias Ivarsson
 tobias.ivars...@neotechnology.com wrote:

  For getting a performance boost out of writes, doing multiple operations
 in
  one transaction will give a much bigger gain than multiple threads
 though.
  For your use case, I think two writer threads and a few hundred elements
 per
  transaction is an appropriate size.

 I got some numbers.

 On a base of 10 rounds of 10 rows each I got an average of
 111.1811 sec to crunch a chunk of data, so it means that it would take
 1.111811 ms to process a single row.
 A single file of data contains data for a single day and has an
 average of 250 rows so it would take approximately 46 minutes to
 crunch.

 The final db size is 588034K (574M) which has 100 rows so we can
 estimate that the final DB size would be 132307650K (126G).

 The current SQL DB is 60G and the app takes 4 and 1/2 hours to crunch
 a month of logs, identical machine.

 The test has been conducted starting from an empty db and the progress
 of chunk time is this one: 80307 ms, 83444 ms, 97162 ms, 131703 ms,
 134647 ms, 104602 ms, 115944 ms, 112489 ms, 115660 ms, 135853 ms.

 There are 20 threads (default config) and they're processing 200 rows
 each within a single Transaction.

 The JVM is:
 OpenJDK Runtime Environment (build 1.6.0-b20) and was started with this
 options:
 -Djava.awt.headless=true -Xms40m -Xmx1536m -XX:+UseConcMarkSweepGC

 And ...

 What do you think?

 Thanks for any info you could give
 --
 Massimo
 http://meridio.blogspot.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] neo4j 1.1 Download? -- for Use with Current Python Bindings

2011-02-01 Thread Tobias Ivarsson
Neo4j 1.1 can be downloaded here:
http://neo4j.org/get?file=neo4j-apoc-1.1.tar.gz

On Tue, Feb 1, 2011 at 3:59 AM, James Thornton ja...@jamesthornton.comwrote:

 Since the current python bindings only work with neo4j 1.1 (as per
 this bug report https://trac.neo4j.org/ticket/302 and
 http://wiki.neo4j.org/content/Getting_Started_With_Python), where can
 you download neo4j 1.1 in the meantime until the python bindings are
 updated?
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Help with exception

2011-02-01 Thread Tobias Ivarsson
Are threads being interrupted anywhere in your application?
There is an issue in java.nio that it closes open channels if interrupted
during an operation.
This means that io-channels that are shared between multiple threads gets
closed if one of the threads using it gets interrupted.

Or does this happen with a freshly started application?
If you could send over the messages.log file that might shed some light on
what has happened to your store.

Cheers,
-tobias

On Tue, Feb 1, 2011 at 4:43 PM, Paul A. Jackson paul.jack...@pb.com wrote:

 I've been doing some performance and scalability testing with large graphs
 (2,000,000 nodes, 5,000,000 edges - actually, the WikiTalk data from the
 Stanford Snap site). I must have shut down my server improperly because a
 number of graphs needed to recover when I started it back up, but this
 largest of the graphs didn't appear to recover properly. I now get the
 following exception when I attempt to load data from the graph. I am still
 using neo4j 1.1.

 Can anyone say what the exception means? Corrupt database, throw it away
 and rebuild it?

 Exception in thread com.g1.dcg.graph.neo4j.NeoEigenvectorJob:2
 java.lang.RuntimeException:
 org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException: Unable to
 load position[146166] @[1315494]
 at com.g1.dcg.graph.job.AbstractGraphJob.run(AbstractGraphJob.java:59)
 at java.lang.Thread.run(Thread.java:619)
 Caused by: org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException:
 Unable to load position[146166] @[1315494]
 at
 org.neo4j.kernel.impl.nioneo.store.PersistenceRow.readPosition(PersistenceRow.java:101)
 at
 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:152)
 at
 org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:474)
 at
 org.neo4j.kernel.impl.nioneo.store.NodeStore.loadLightNode(NodeStore.java:131)
 at
 org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.nodeLoadLight(ReadTransaction.java:74)
 at
 org.neo4j.kernel.impl.nioneo.xa.NioNeoDbPersistenceSource$ReadOnlyResourceConnection.nodeLoadLight(NioNeoDbPersistenceSource.java:235)
 at
 org.neo4j.kernel.impl.persistence.PersistenceManager.loadLightNode(PersistenceManager.java:74)
 at
 org.neo4j.kernel.impl.core.NodeManager.getNodeById(NodeManager.java:391)
 at
 org.neo4j.kernel.EmbeddedGraphDbImpl.getNodeById(EmbeddedGraphDbImpl.java:223)
 at
 org.neo4j.kernel.EmbeddedGraphDbImpl$AllNodesIterator.hasNext(EmbeddedGraphDbImpl.java:426)
 at
 com.g1.dcg.graph.neo4j.NeoEigenvectorJob.runJob(NeoEigenvectorJob.java:72)
 at com.g1.dcg.graph.job.AbstractGraphJob.run(AbstractGraphJob.java:51)
 ... 1 more
 Caused by: java.nio.channels.ClosedChannelException
 at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88)
 at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:299)
 at
 org.neo4j.kernel.impl.nioneo.store.PersistenceRow.readPosition(PersistenceRow.java:80)
 ... 12 more


 Thanks,
 -Paul
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Help with exception

2011-02-01 Thread Tobias Ivarsson
Oh, sorry, I missed that you were still using 1.1, messages.log was
introduced after 1.1 was released.

However, the exception you supplied does not say that the store is corrupt,
just that it cannot read it, because it has been closed.
There are only two scenarios I can think of where this can happen:
1. Concurrent invocation of grapdb.shutdown()
2. A thread was interrupted while doing IO operations on that same file.

Any chance your code could have already called graphdb.shutdown()?
For example a main() that starts a bunch of threads then doesn't wait for
them to finish but instead shuts down the graph database?

Cheers,
Tobias

On Tue, Feb 1, 2011 at 5:09 PM, Paul A. Jackson paul.jack...@pb.com wrote:

 This happens with a freshly started database.  I don't see a file with the
 name that you mention.  All I have are the following files with the word log
 in them:

 graph.WikiTalk\active_tx_log
 graph.WikiTalk\nioneo_logical.log.1
 graph.WikiTalk\nioneo_logical.log.active
 graph.WikiTalk\tm_tx_log.1
 graph.WikiTalk\tm_tx_log.2
 graph.WikiTalk\lucene\lucene.log.1
 graph.WikiTalk\lucene\lucene.log.active
 graph.WikiTalk\lucene-fulltext\lucene.log.1
 graph.WikiTalk\lucene-fulltext\lucene.log.active

 Thanks,
 -Paul

 -Original Message-
 From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org]
 On Behalf Of Tobias Ivarsson
 Sent: Tuesday, February 01, 2011 10:58 AM
 To: Neo4j user discussions
 Subject: Re: [Neo4j] Help with exception

 Are threads being interrupted anywhere in your application?
 There is an issue in java.nio that it closes open channels if interrupted
 during an operation.
 This means that io-channels that are shared between multiple threads gets
 closed if one of the threads using it gets interrupted.

 Or does this happen with a freshly started application?
 If you could send over the messages.log file that might shed some light on
 what has happened to your store.

 Cheers,
 -tobias

 On Tue, Feb 1, 2011 at 4:43 PM, Paul A. Jackson paul.jack...@pb.com
 wrote:

  I've been doing some performance and scalability testing with large
 graphs
  (2,000,000 nodes, 5,000,000 edges - actually, the WikiTalk data from the
  Stanford Snap site). I must have shut down my server improperly because a
  number of graphs needed to recover when I started it back up, but this
  largest of the graphs didn't appear to recover properly. I now get the
  following exception when I attempt to load data from the graph. I am
 still
  using neo4j 1.1.
 
  Can anyone say what the exception means? Corrupt database, throw it away
  and rebuild it?
 
  Exception in thread com.g1.dcg.graph.neo4j.NeoEigenvectorJob:2
  java.lang.RuntimeException:
  org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException: Unable to
  load position[146166] @[1315494]
  at
 com.g1.dcg.graph.job.AbstractGraphJob.run(AbstractGraphJob.java:59)
  at java.lang.Thread.run(Thread.java:619)
  Caused by: org.neo4j.kernel.impl.nioneo.store.UnderlyingStorageException:
  Unable to load position[146166] @[1315494]
  at
 
 org.neo4j.kernel.impl.nioneo.store.PersistenceRow.readPosition(PersistenceRow.java:101)
  at
 
 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:152)
  at
 
 org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:474)
  at
 
 org.neo4j.kernel.impl.nioneo.store.NodeStore.loadLightNode(NodeStore.java:131)
  at
 
 org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.nodeLoadLight(ReadTransaction.java:74)
  at
 
 org.neo4j.kernel.impl.nioneo.xa.NioNeoDbPersistenceSource$ReadOnlyResourceConnection.nodeLoadLight(NioNeoDbPersistenceSource.java:235)
  at
 
 org.neo4j.kernel.impl.persistence.PersistenceManager.loadLightNode(PersistenceManager.java:74)
  at
  org.neo4j.kernel.impl.core.NodeManager.getNodeById(NodeManager.java:391)
  at
 
 org.neo4j.kernel.EmbeddedGraphDbImpl.getNodeById(EmbeddedGraphDbImpl.java:223)
  at
 
 org.neo4j.kernel.EmbeddedGraphDbImpl$AllNodesIterator.hasNext(EmbeddedGraphDbImpl.java:426)
  at
 
 com.g1.dcg.graph.neo4j.NeoEigenvectorJob.runJob(NeoEigenvectorJob.java:72)
  at
 com.g1.dcg.graph.job.AbstractGraphJob.run(AbstractGraphJob.java:51)
  ... 1 more
  Caused by: java.nio.channels.ClosedChannelException
  at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88)
  at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:299)
  at
 
 org.neo4j.kernel.impl.nioneo.store.PersistenceRow.readPosition(PersistenceRow.java:80)
  ... 12 more
 
 
  Thanks,
  -Paul
  ___
  Neo4j mailing list
  User@lists.neo4j.org
  https://lists.neo4j.org/mailman/listinfo/user
 



 --
 Tobias Ivarsson tobias.ivars...@neotechnology.com
 Hacker, Neo Technology
 www.neotechnology.com
 Cellphone: +46 706 534857
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https

Re: [Neo4j] Lucene index commit rate and NoSuchElementException

2011-02-01 Thread Tobias Ivarsson
Since you are checking for existence before inserting the conflict you are
getting is strange. Are you running multiple insertion threads?

-Tobias

On Tue, Feb 1, 2011 at 6:19 PM, Massimo Lusetti mluse...@gmail.com wrote:

 Hi everyone,
  I'm new to neo4j and I'm making experience with it, I got a fairly
 big table (in my current db) which consists of something more then 220
 million rows.

 I want to put that in a graphdb, for instance neo4j, and graph it to
 do some statistics on them. Every row will be a node in my vision with
 just one property and will have an index to check if cause I'm using
 small chunk of data to test and want to know if I've already imported
 that specific row.

 I'm pretty sure there aren't two identical rows in the DB table due to
 db constraints but I still get this errors as import proceed:
 java.util.NoSuchElementException: More than one element in
 org.neo4j.index.impl.lucene.LuceneIndex$1@77435b. First element is
 'Node[201728]' and the second element is 'Node[201744]'

 Does this has something to do with the commit rate of the neo4j
 internal Lucene index? Or I'm doing something wrong? BTW I'm
 committing on every row imported:

  String rowMD5 = md5Source.digest(logRow.getRaw().getBytes());

Transaction tx = graphDb.beginTx();

IndexNode md5Index = graphDb.index().forNodes(md5-rows);

if (md5Index.get(md5, rowMD5).getSingle() == null)
{
Node rowNode = graphDb.createNode();
String[] logRowArray = LogRow.asStringArray(logRow);
rowNode.setProperty(logRow, logRowArray);
md5Index.add(rowNode, md5, rowMD5);

tx.success();
}

tx.finish();


 Thanks for any clue...

 Regards
 --
 Massimo
 http://meridio.blogspot.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] InvalidRecordException during traversal after unclean shutdown

2011-02-01 Thread Tobias Ivarsson
And this was not using the batchinsert either? Because batchinsert will most
likely leave the data corrupt if not shut down cleanly.

Just want to make sure, by covering the simplest things first.

-tobias

On Tue, Feb 1, 2011 at 7:12 PM, Lars Hvile lars.hv...@fovea.no wrote:

 This exact error has already been discussed on the mailinglist back in 2009
 (http://lists.neo4j.org/pipermail/user/2009-November/001906.html), but
 without a conclusion?

 During some inserts, I did an unclean shutdown. When I restarted neo, the
 standard recovery stuff was logged,
 and it said that recovery was successful, but my data is corrupt, and I
 can't find a way to restore it.
 (this is just test-data, so the actual data is a non-issue, however the
 recovery process is a huge issue for us...)

 Our basic test-structure is a tree, root (not the reference-node) -
 groupA/B/C - nodeA/B/C etc, after the unclean shutdown, somehow
 the root-node has gotten a reference to a relationship on one of the
 leaf-nodes, i.e. nodeA..

 -- directly after successful recovery
 org.neo4j.kernel.impl.nioneo.store.InvalidRecordException: Node[1] is
 neither firstNode[2055881] nor secondNode[2055876] for Relationship[2055394]
at
 org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:133)
at
 org.neo4j.kernel.impl.nioneo.xa.NioNeoDbPersistenceSource$ReadOnlyResourceConnection.getMoreRelationships(NioNeoDbPersistenceSource.java:281)
at
 org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:101)
at
 org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:586)
at
 org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:359)
at
 org.neo4j.kernel.impl.core.IntArrayIterator.hasNext(IntArrayIterator.java:116)
at
 org.neo4j.helpers.collection.NestingIterator.fetchNextOrNull(NestingIterator.java:67)
at
 org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:56)
at
 org.neo4j.kernel.impl.traversal.TraversalBranchImpl.next(TraversalBranchImpl.java:94)
at
 org.neo4j.kernel.PreorderBreadthFirstSelector.next(PreorderBreadthFirstSelector.java:49)
at
 org.neo4j.kernel.impl.traversal.TraverserImpl$TraverserIterator.fetchNextOrNull(TraverserImpl.java:128)
at
 org.neo4j.kernel.impl.traversal.TraverserImpl$TraverserIterator.fetchNextOrNull(TraverserImpl.java:95)
at
 org.neo4j.helpers.collection.PrefetchingIterator.hasNext(PrefetchingIterator.java:56)
at
 org.neo4j.kernel.impl.traversal.OldTraverserWrapper$TraverserImpl.hasNext(OldTraverserWrapper.java:77)
at
 testing.persistence.neo4j.Neo4jEntityStore.traverse(Neo4jEntityStore.java:154)
at testing.Launcher$3.execute(Launcher.java:74)
at
 testing.persistence.neo4j.Neo4jEntityRepo.access(Neo4jEntityRepo.java:116)
at testing.Launcher.doStuff(Launcher.java:72)
at testing.Launcher.main(Launcher.java:32)

 -- after manual deletion of Relationship[2055394], in a desperate
 recovery-attempt =)
 org.neo4j.kernel.impl.nioneo.store.InvalidRecordException: Node[1] is
 neither firstNode[0] nor secondNode[0] for Relationship[2055394]
at
 org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:133)
at
 org.neo4j.kernel.impl.nioneo.xa.NioNeoDbPersistenceSource$ReadOnlyResourceConnection.getMoreRelationships(NioNeoDbPersistenceSource.java:281)
at
 org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:101)
at
 org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:586)
at
 org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:359)
at
 org.neo4j.kernel.impl.core.IntArrayIterator.hasNext(IntArrayIterator.java:116)
at
 testing.persistence.neo4j.Neo4jEntityStore.traverse(Neo4jEntityStore.java:150)
at testing.Launcher$3.execute(Launcher.java:74)
at
 testing.persistence.neo4j.Neo4jEntityRepo.access(Neo4jEntityRepo.java:116)
at testing.Launcher.doStuff(Launcher.java:72)
at testing.Launcher.main(Launcher.java:32)

 Unfortunately I don't have a copy of the database at it's original corrupt
 state, I do however have a copy of it after I deleted the the relationship..
 788MiB zipped


 Regards,
 Lars Hvile
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


Re: [Neo4j] Lucene index commit rate and NoSuchElementException

2011-02-01 Thread Tobias Ivarsson
No, it means that you have to synchronize the threads so that they don't
insert the same data concurrently.
Perhaps a ConcurrentHashMapMD5,token where you would putIfAbsent(md5,new
Object()) when you start working on a new hash. If the token Object you get
back is not the same as you put in, you know that another thread is working
on that md5, which means this thread should move on to another one. When the
transaction is done you remove the md5 from the Map, to ensure that you
don't leak memory.

That's a simple locking on arbitrary key implementation. The reason you
cannot just do synchronized(md5) {...} is of course that your hashes are
computed, and thus will not be the same object every time, even though they
are equals().

For getting a performance boost out of writes, doing multiple operations in
one transaction will give a much bigger gain than multiple threads though.
For your use case, I think two writer threads and a few hundred elements per
transaction is an appropriate size.

-tobias

On Tue, Feb 1, 2011 at 9:06 PM, Massimo Lusetti mluse...@gmail.com wrote:

 On Tue, Feb 1, 2011 at 8:02 PM, Mattias Persson
 matt...@neotechnology.com wrote:

  Seems a little weird, the commit rate won't affect the end result,
  just performance (more operations per commit means faster
  performance). Your code seems correct for single threaded use btw.

 Does it means that I cannot access the graphdb from multiple threads?
 That code is on a singleton service which expose the
 GraphDatabaseService through a method addNode() from where I run that
 code.

 The singleton service is called by a thread pool which can fire at
 maximum 20 concurrent threads.

 Any hints is really appreciated.

 Cheers
 --
 Massimo
 http://meridio.blogspot.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user




-- 
Tobias Ivarsson tobias.ivars...@neotechnology.com
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user


  1   2   3   4   >