Re: [Neo4j] Cypher DSL and result mapping
On Fri, Nov 4, 2011 at 10:45 AM, Rickard Öberg < rickard.ob...@neotechnology.com> wrote: > On 11/4/11 17:29 , Daniel Yokomizo wrote: > >> What are the projection methods that you would like to see? What are > >> common usecases? > > > > Supporting @java.beans.ConstructorProperties is nice because we can use > > immutable objects as dtos. > > Right, that is a good point, and that will definitely be supported. > +1, a really good suggestion. I'm not a big fan of that annotation per se, it would have been nicer to annotate each parameter individually, but it is standardized which makes it VERY compelling. -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 (Swe); +1 650 450 3806 (US) ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Cypher DSL and result mapping
I started a prototype for something similar a while back: https://github.com/thobe/neo4j-community-experiments/tree/cypher-annotations It has a different approach to constructing queries, it uses annotations with query strings in them, but uses compile time annotation processing to process those annotations in order to verify syntax. In the future (when Cypher supports it) I'd like to do type checking as well. What it has that is similar though is the way it converts the result to java classes. I know that Spring-Data-Neo4j does similar things as well. Since that kind of conversion seems to be something that multiple higher level libraries do, it would probably make sense to move it to a shared place. Note please that this branch is just three hours worth of throwing some ideas together, it's not complete, but should at least display the ideas. -tobias On Fri, Nov 4, 2011 at 6:10 AM, Rickard Öberg < rickard.ob...@neotechnology.com> wrote: > Hi, > > I've briefly started to look at how to map Cypher query results into > DTO's. I'm using the QueryDSL helper classes to do it, but instead of > putting the projection methods into the query object (as QueryDSL does > it), I have a separate Projection class for it. The reason is that I > don't want to require DSL users to have access to the execution engine, > which will obviously not be the case if you are creating queries on the > client which are to be sent to the server. > > Here's an example of what I want it to look like when used: > Projection projection = new Projection(engine); > > Iterable friends = projection.iterable( start( node( "john", > john ) ) > .match( path().from( "john" ).out( "friend" ) > .link().out( "friend" ).to( "fof" ) ) > .returns( properties( "john.name", "fof.name" ) ), > Friend.class ); > System.out.println( friends ); > --- > So the Projection encapsulates the reference to the query execution, > which means that you don't need to have a reference to it when executing > the queries to do the projection. The "iterable" method takes a DSL > query and the Java class you want the results to be projected into. > Pretty straightforward. > > For now the code assumes that the fields in Friend is the same as the > order of the return results. This is broken, and will be fixed later to > use proper mapping, preferably with aliases in the return statement as > hints. > > What are the projection methods that you would like to see? What are > common usecases? > > /Rickard > > -- > Rickard Öberg > Developer > Neo Technology > Twitter: @rickardoberg, Skype: rickardoberg > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 (Swe); +1 650 450 3806 (US) ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Article: "The Coming SQL Collapse"
We had an interesting discussion about this internally at Neo Technology today. We thought it might be of interest to the broader community. I don't think the discussion is over, so it would be interesting to continue it on the public mailing list. It regards the initial paragraphs of an article posted to dzone recently: http://www.dzone.com/links/rss/the_coming_sql_collapse.html It mentions Neo4j and how the author dislikes a common way of using Neo4j for building applications. It would be interesting to hear suggestions on how to improve this. Forwarded conversation follows: On Fri, Oct 14, 2011 at 10:13 AM, Tobias Ivarsson < tobias.ivars...@neotechnology.com> wrote: > I found this while reading feeds in bed last night: > > *The Coming SQL Collapse* > http://www.dzone.com/links/rss/the_coming_sql_collapse.html > > (Sent from Flipboard <http://flipboard.com>) > > > The things he say about SQL vs NOSQL is not very interesting, but I'd like > to raise what he says about Neo4j: > > I looked at neo4j briefly the other day, and quite predictably thought >> ‘wow, this looks like a serious tinkertoy: it‘s basically a bunch of nodes >> where you just blob your attributes.‘ Worse than that, to wrap objects >> around it, you have to have them explicitly incorporate their node class, >> which is ugly, smelly, violates every law of separation of concerns and >> logical vs. physical models. On the plus side, as I started to look at it >> more, I realized that it was the perfect way to implement a backend for a >> bayesian inference engine (more on that later). Why? Because inference >> doesn‘t care particularly about all the droll requirements that are settled >> for you by SQL, and there are no real set operations to speak of. > > > He attacks our pattern of building domain models with Neo4j, calling it > "ugly", "smelly" and "in violation of every law of separation of concerns > and logical vs. physical models". Is he right? My feeling is that he is > brain washed with too many so called "best practices", but Neo4j has been my > main model for a long time now, my perspective is likely skewed. I'd like to > hear your thoughts. > > -- > Tobias Ivarsson > On Fri, Oct 14, 2011 at 10:32 AM, Rickard Öberg < rickard.ob...@neotechnology.com> wrote: > > > Well, I'd tend to agree with the author. Mixing persistence details with > the domain model itself is really a bad idea. Infrastructure details should > not pollute the domain logic as it does with the currently "suggested" usage > of Neo4j. But I think both Spring Data Graph and the Qi4j usage model fixes > this, as it allows you to keep many of those things outside of the domain > code. > > /Rickard On Fri, Oct 14, 2011 at 11:45 AM, Tobias Ivarsson wrote: > On Fri, Oct 14, 2011 at 11:21 AM, Rickard Öberg < > rickard.ob...@neotechnology.com> wrote: > >> On 10/14/11 17:16 , Tobias Ivarsson wrote: >> >>> I was hoping for a bit more elaboration, of why it is a bad idea. >>> >>> Spring Data Neo4j operates mainly in the same way (at least it did when >>> I was part of the design process), it just hides the details of it. >>> >>> The model we suggest is not to mix infrastructure details (nodes, >>> relationships, traversals) with the domain logic. We suggest the domain >>> logic be a separate layer, acting on domain data objects (defined as a >>> set of interfaces). What we do suggest though is for those domain data >>> objects to be implemented as wrappers of nodes and relationships. >>> >> >> That sounds like "transaction script+anemic domain model", which is an >> anti-pattern as well. > > > I'm guessing the anemic domain model is what you are pointing out as the > anti-pattern. I can see how transaction scripts are and ADM usually come > together though. > > References: > > http://martinfowler.com/eaaCatalog/transactionScript.html > http://martinfowler.com/bliki/AnemicDomainModel.html > > > >> Domain logic should be in the domain objects, and so splitting them into >> several layers confuse things more than it helps. > > > I agree with you, SDN "solves" this, and does so well. > > >> So is the bad part of it just the part of having the implementation of >>> your domain data objects deal with the Neo4j API, instead of having DTOs >>> and DAOs that you load from the database, operate on in memory, then >>> store back to the db at explicit points? Or are there deeper issues than >>> that? >>> >> >> It relates to testability (you
Re: [Neo4j] Clarification on read and write locks
On Mon, Sep 26, 2011 at 2:28 AM, Bryce wrote:
> Is there any reason that access to the lock manager is a little difficult?
> Is there any issue with using it (both single server and HA)? Are their
> any issues to look out for? And does the above code look workable?
>
The reason the LockManager is "hidden away" is because it is considered an
implementation detail.
We reserve the right to change it at any point without prior notice.
>From one version to another, the API of this class could potentially look
completely different, or have very different semantics.
If you write code that depend on implementation internal classes it will at
best cause you compilation errors when upgrading to a later version of
Neo4j, at worst break in subtle ways after you've deployed your application.
There is another important thing to note about your sample code:
On Mon, Sep 26, 2011 at 2:28 AM, Bryce wrote:
>
>private void acquireLock( LockType lockType )
>{
>GraphDatabaseService graphDb = baseNode.getGraphDatabase();
>if ( lockType == LockType.READ && graphDb instanceof
> AbstractGraphDatabase )
>{
>((AbstractGraphDatabase)
> baseNode.getGraphDatabase()).getConfig().getLockManager().getReadLock(
> baseNode );
>}
>else {
>// default to write lock if read locks unavailable
>baseNode.removeProperty( "___dummy_property_to_acquire_lock___"
> );
>}
>}
>
The two different code paths in this code have very different semantics. The
first path (if it is a read lock and the db is and AbstractGraphDatabase)
will acquire a read lock and hold it indefinitely. The second path will try
to remove a dummy property, resulting in a write lock that gets tied to the
current transaction, and thus gets released when the transaction completes.
There is another class involved in the lock management in Neo4j, it is
called LockReleaser. All locks that are acquired through the public APIs are
registered with the LockReleaser. This will tie the lock to the transaction
to ensure that the lock is released when the transaction completes. If the
lock is only acquired through the LockManager, and not registered with the
LockReleaser, you need to manually release the lock.
The reason for the remove-non-existing-property-hack is that we haven't
decided on a good public API for lock management. The questions around this
are:
* Should there be release() methods for locks as well? or should locks
always be tied to the transaction?
* Where should the methods live? On the entities themselves? i.e.
node.acquireReadLock() / node.aquireWriteLock() or on the Transaction object
to signal that locks are tied to the life cycle of the transaction? i.e.
tx.readLock(node) / tx.writeLock(node)
Cheers,
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857 (Swe); +1 650 450 3806 (US)
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Synchronization of EmbeddedReadOnlyGraphDatabase - Bug?
I think a bit of elaboration might be in order. EmbeddedReadOnlyGraphDatabase was created for one specific purpose: Being able to interactively introspect a graph without having to shut down the application that uses it. Specifically the tools that we wanted to support with this were the Neo4j shell and Neoclipse. EmbeddedReadOnlyGraphDatabase (EROGD) has two major issues with way caching is done internally in Neo4j (one issue with each cache): - When the EROGD reads data from the file system it will, like a normal EGD, cache the node and relationship objects. If a normal EGD modifies the graph "under the feet" of the EROGD, there is no way for the EROGD to know that the data in cache is now stale, which will lead to an inconsistent view of the graph. If for example the EROGD has cached Node[15] with the information that it is connected to some other node through Relationship[344], and Relationship[344] is deleted you will get InvalidRecordException (as you described). And of course if relationships are added to Node[15] these will not be seen at all by the EROGD (until Node[15] is evicted from the cache due to not being used for a while). - Neo4j also caches data on the filesystem level by memory mapping (mmap) hot regions of the store files. Writes to these regions will not be flushed to the actual file until the mmapped window is evicted due to being less hot than other windows, or when the transaction log for Neo4j is rotated. This means that from the p.o.v. of the EROGD the actual data written to disk will look inconsistent. Which would also lead to InvalidRecordExcaption. This situation is actually made even more complicated by the fact that unix operating systems will attempt to share memory mapped data from the same file between multiple processes, but the normal EGD and the EROGD will not make the same decisions on which regions to mmap, they might not even decide on the same size for mmap windows. We haven't tested how well different operating systems deal with reading data that was written to an mmap region through non-mmap syscalls from a different process, most likely this varies from OS to OS. The second of these problems is of course the worst, since it cannot be worked around. The first one can be mitigated by configuring Neo4j to not use the object cache, by passing the cache_type=none parameter to the constructor of the EROGD. This should really be made default for EROGD, unless we decide to completely remove EROGD. I hope that sheds some light on the reasons why you experience these problems with EmbeddedReadOnlyGraphDatabase, and what the intention of creating it was. As a side note I can mention that I had a different idea for how to solve the introspection-of-live-graph problem at the time EmbeddedReadOnlyGraphDatabase was created: Create network based implementation of the GraphDatabaseService API and connect directly to the running instance. This would completely avoid the cache staleness problem, but at the cost of network overhead for each graph operation, which is probably fine for tooling purposes. With the JVM agent attach protocol it would be possible to inject such a server into a running graph database that wasn't originally configured for it. I in fact implemented this as the RemoteGraphDatabase subproject. Since my colleagues did not share my vision about that idea, this project didn't receive much attention after its initial inception. It was also never really used for these purposes, but rather misused for building applications, leading us to deprecate the project. When we then later discovered a severe bug in the implementation of the remote transaction handling logic, we completely removed the project. I still believe this to be a superior model for tools, but would build it differently if I were to build it today. -tobias On Mon, Aug 1, 2011 at 4:48 PM, Jim Webber wrote: > Hi Mathias, > > EmbeddedReadOnlyGraphDatabase is not quite what it seems, and I think > should be deprecated/removed. The correct way for database instances to > become consistent is through the HA protocol. > > Jim > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] cypher packages
They should be in the neo4j-cypher-1.4.jar. That jar-file should be among your dependencies if you simply depend on (in maven): org.neo4j neo4j 1.4 Cheers, Tobias On Thu, Jul 28, 2011 at 12:07 PM, Jean-Sébastien Stoffen wrote: > Hi, > > Sorry for the newbie questions who happens very frequently at the begining. > > I'm trying to submit some request to my graph using the cypher query > language > > I get those messages : > unable to resolve class CypherParser > unable to resolve class ExecutionResult > ... > > when I try the example showed here : > http://docs.neo4j.org/chunked/1.4.M04/query-lang.html > > After some research, I've not found where I can download those packages > I've searched for org.neo4j.cypher.javacompat but without succes. > > Could you tell me where to find it? > > Regards, > Jean-Sébastien Stoffen. > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Django + Neo4j seamless installation error: No module named CallbackFilter
ango documentation: https://docs.djangoproject.com/en/dev/topics/logging/#id3 the CallbackFilter feature seems to not be released yet. How it got into your settings.py (I'm guessing that is where 'require_debug_false' is configures) I don't know. If I had not had your complete description of everything you did, I would have guessed that you either: a) Followed the documentation of the wrong version of Django from their website (the dev version is easier to find then the ones for released versions). Or b) Used a newer version of Django to auto-generate the settings.py file, and then an older version to run the code. But since neither of those steps are in your description, my best guess is that the Django folks released parts of that feature too early. You'd think there be more mentionings of this problem online then though. Do you think you could grep through your settings.py for CallbackFilter and/or require_debug_false to see if that might be the problem. Cheers, -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] No index provider 'lucene' found
at java.lang.Class.newInstance0(Class.java:355) >at java.lang.Class.newInstance(Class.java:308) >at org.mortbay.jetty.servlet.Holder.newInstance(Holder.java:153) >at > org.mortbay.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:339) >at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:463) >at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:362) >at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) >at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:729) >at > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) >at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >at > org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49) >at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >at org.mortbay.jetty.Server.handle(Server.java:324) >at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505) >at > org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:843) >at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:647) >at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380) >at > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395) >at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488) > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Generic expansion?
org.neo4j.kernel.Traversal.expanderForAllTypes();
On Fri, May 20, 2011 at 4:53 PM, Marko Rodriguez wrote:
> Hey,
>
> Yea. Thats what I'm looking for, but it doesn't seem to be part of the API?
> .. ?
>
> I can only find StandardExpander (abstract) and OrderedByTypeExpander.
>
> Thoughts?, -- IM me if you wish.
> Marko.
>
> On May 20, 2011, at 8:46 AM, Peter Neubauer wrote:
>
> > Marko,
> > you could do an AllExpander(Direction.BOTH) or really any subclass of
> > Expander, http://api.neo4j.org/current/org/neo4j/graphdb/Expander.html
> >
> > Does that help?
> >
> > Cheers,
> >
> > /peter neubauer
> >
> > GTalk: neubauer.peter
> > Skype peter.neubauer
> > Phone +46 704 106975
> > LinkedIn http://www.linkedin.com/in/neubauer
> > Twitter http://twitter.com/peterneubauer
> >
> > http://www.neo4j.org - Your high performance graph
> database.
> > http://startupbootcamp.org/- Öresund - Innovation happens HERE.
> > http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.
> >
> >
> > On Fri, May 20, 2011 at 4:43 PM, Marko Rodriguez >wrote:
> >
> >> Hi,
> >>
> >> How do I do something like this:
> >>
> >> PathFinder finder = GraphAlgoFactory.shortestPath(new
> >> OrderedByTypeExpander().add(DynamicRelationshipType.withName("knows")),
> 6);
> >>
> >> ...but where I don't care about the RelationshipType taken (and I don't
> >> want to enumerate all the types I have as I have 100's of types).
> >>
> >> Above, when I don't have "knows," it doesn't find any paths.
> >>
> >> Thanks,
> >> Marko.
> >>
> >> http://markorodriguez.com
> >> ___
> >> Neo4j mailing list
> >> User@lists.neo4j.org
> >> https://lists.neo4j.org/mailman/listinfo/user
> >>
> > ___
> > Neo4j mailing list
> > User@lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] about relationship whit himself.
Neo4j does not support relationships with the same start node as end node yet. This is being worked on however. We actually had it working, passing even the meanest test cases we could come up with, for a few minutes last friday. But then we were given a choice between the functionality of these self relationships, or a substantial performance improvement. We went for the latter, putting self relationships on hold for a while. We have some ideas that could make it possible to put self relationships back in, but I don't know when we can get there. Cheers, Tobias On Wed, May 18, 2011 at 8:54 PM, Jose Angel Inda Herrera < jai...@estudiantes.uci.cu> wrote: > hello list, > I have a question you would like to clarify, > in a graph may have a sibling with himself, that I have it clear. > neo4j now allow this??, because when I create a relation from ni to nf, > where ni = nf gives me a exetion, > > thanks very much, > cheers. > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Color suggestions for the Self-Relationship bike shed
Having an API that changes behavior based on configuration is a very bad idea. It makes the system very hard to work with. On Tue, May 17, 2011 at 8:30 PM, Craig Taverner wrote: > What about a system config enabling/disabling loops? Then we could have > option 1, but for people that never loops, they can still get the extra > loop > check by setting the system config option. > > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] GraphRepository parameter
Hi, Yes, it has to be a special type. Your class has to extend GraphBacked (as clearly stated by the error you are getting). The AspectJ compiler will make your class extend GraphBacked automatically if you have annotated it with either NodeEntity or RelationshipEntity. The error you are experiencing will usually occur when AspectJ is not present, are you sure you have set up AspectJ properly in your build pipeline. Cheers, Tobias On Tue, May 17, 2011 at 2:48 PM, mujer_esponja wrote: > Hello!! > I have asked another question just a few hours ago, but I am new on neo4j > and I need some help. > > Now the problem is with this class > > http://static.springsource.org/spring-data/data-graph/docs/current/api/org/springframework/data/graph/neo4j/repository/GraphRepository.html > > In a declaration I need to do: GraphRepository > as I have seen in examples. > > But I am getting the error: > > Bound mismatch: The type MyClass is not a valid substitute for the bounded > parameter > of the type > GraphRepository > > Must it be a special type? Where is the error?? Thanks in advance > > -- > View this message in context: > http://neo4j-user-list.438527.n3.nabble.com/GraphRepository-parameter-tp2952354p2952354.html > Sent from the Neo4J User List mailing list archive at Nabble.com. > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] ClosedChannelExceptions in highly concurrent environment
Hi Jennifer,
Could you reproduce it on your side by doing the same kind of systems tests
again? If you could then I'd be very happy if you could try a patched
version that we have been working on and see if that fixes the issue.
Cheers,
Tobias
On Tue, May 17, 2011 at 2:49 AM, Jennifer Hickey wrote:
> Hi Tobias,
> Unfortunately I don't have an isolated test case, as I was doing a fairly
> involved system test at the time. I may be able to have a colleague work on
> reproducing it at a later date (I've been diverted to something else for the
> moment).
>
> I was remote debugging with Eclipse, so I toggled a method breakpoint on
> Thread.interrupt() and then inspected the stack once the breakpoint was hit.
>
> Sorry I don't have more information at the moment. I agree that
> eliminating the interrupts sounds like the best approach, if possible.
>
> Thanks,
> Jennifer
>
> From: user-boun...@lists.neo4j.org [user-boun...@lists.neo4j.org] On
> Behalf Of Tobias Ivarsson [tobias.ivars...@neotechnology.com]
> Sent: Thursday, April 28, 2011 6:23 AM
> To: Neo4j user discussions
> Subject: Re: [Neo4j] ClosedChannelExceptions in highly concurrent
> environment
>
> Hi Jennifer,
>
> I'd first like to thank you for the testing and analysis you've done. Very
> useful stuff. Do you think you could send some test code our way that
> reproduces this issue?
>
> This is actually the first time this issue has been reported, so I wouldn't
> say it is a common issue. My guess is that your thread volume triggered a
> rare condition that wouldn't be encountered otherwise.
>
> I'm also curious to know how you found the source of the interruptions.
> When
> I debug thread interruptions I've never been able to find out where the
> thread got interrupted from without doing tedious procedures of breakpoint
> +
> logging + trying to match thread ids. If you have a better method for doing
> that I'd very much like to know.
>
> I think we should focus the effort on fixing the interruption issue if we
> can. And I believe we would be able to do that if the interruptions do in
> fact originate from where you say they do. But the suggestion of being able
> to switch the lucene directory implementation is still interesting, but as
> you point out since it has issues on some platforms it would be better if
> we
> could be rid of the interruption issue.
>
> Cheers,
> Tobias
>
> On Thu, Apr 28, 2011 at 12:41 AM, Jennifer Hickey >wrote:
>
> > Hello,
> > I've been running some tests w/approx 400 threads reading various indexed
> > property values. I'm running on 64 bit Linux. I was frequently seeing
> the
> > ClosedChannelException below. The javadoc on Lucene's NIOFSDirectory
> states
> > that "Accessing this class either directly or indirectly from a thread
> while
> > it's interrupted can close the underlying file descriptor immediately if
> at
> > the same time the thread is blocked on IO. The file descriptor will
> remain
> > closed and subsequent access to {@link NIOFSDirectory} will throw a
> {@link
> > ClosedChannelException}. If your application uses either {@link
> > Thread#interrupt()} or {@link Future#cancel(boolean)} you should use
> {@link
> > SimpleFSDirectory} in favor of {@link NIOFSDirectory}."
> >
> > A bit of debugging revealed that the Thread.interrupts were coming from
> > Neo4j, specifically in RWLock and MappedPersistenceWindow. So it seems
> like
> > this would be a common problem, though perhaps I am missing something?
> >
> > SimpleFSDirectory seems a bit of a performance bottleneck, so I switched
> to
> > MMapDirectory and the problem did go away. I didn't see a way to switch
> > implementations w/out modifying neo4j code, so I changed LuceneDataSource
> as
> > follows:
> >
> > static Directory getDirectory( String storeDir,
> >IndexIdentifier identifier ) throws IOException
> > {
> >MMapDirectory dir=new MMapDirectory(getFileDirectory( storeDir,
> > identifier), null);
> >if(MMapDirectory.UNMAP_SUPPORTED) {
> >dir.setUseUnmap(true);
> >}
> >return dir;
> > }
> >
> > So I'm wondering if others have seen this problem and/or if there is a
> > recommended solution? Our product runs on quite a few different
> operating
> > systems, so I have some reservations about using MMapDirectory as well
> > (javadoc speaks of a few caveats on Windows, 64 vs 32, etc). Also, I'd
> >
Re: [Neo4j] Color suggestions for the Self-Relationship bike shed
Does anyone NOT planning to use loops have an opinion in the matter? That would be very valuable input. Cheers, -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Color suggestions for the Self-Relationship bike shed
There has been a few discussions around supporting Loops, relationships with the same start node as end node, recently. Both here on the mailing list, and even more at the Neo Technology office. We have a working patch for handling loops in Neo4j, but one final piece is missing: what should the API look like for creating a loop? There are three suggestions for this, I'll list them with the pros and cons we've found for each. It would be great if you could provide some feedback on what you think on the matter, and which alternative you prefer. The alternatives: 1. let the existing createRelationshipTo(Node,RelationshipType) create loops if the same node is passed as argument. 2. add a new createLoop(RelationshipType) method for creating loops. 3. add a createRelationshipOrLoop(Node,RelationshipType) method that would work like createRelationshipTo, except it would permit creating loops. The pros and cons: PRO 1: does not add a new method to the API that developers have to learn. CON 1: changes the semantics of the createRelationshipTo method slightly from what it is today. CON 1: will not help you catch programming errors where you've mistakenly written code that creates a relationship to the same node (most of the cases where code creates relationships to the wrong node it is to the same node). PRO 2: will let you be explicit of when creating a loop. PRO 2: will let createRelationshipTo preserve the semantics it has today. Which will help catching many create relationships to the wrong node cases. CON 2: will force you to be explicit about loops, most applications that wan't loops will just treat them as any relationship where the start node and end node will just happen to be the same. PRO 3: adds loops as a generic construct (start and end just happens to be the same) without changing the current semantics of createRelationshipTo CON 3: Introduces a new method that creates relationships between any two nodes. It would of course be possible to go with both 2 and 3, and I think option 3 makes more sense as an addition to option 2, rather than as an alternative to it. What do you think? -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4j and Java 1.5
The service code was written before Java 1.5 support was dropped, so that will still work. There are a few other places where we use java 6 APIs, mainly java.util.Queue and java.util.concurrent.TimeUnit. javax.scripting is super simple to support under 1.5 (there is a backport package), and it is only used in the server. Java 1.5 is not supported by Neo4j, since it has reached its End Of Life. If really need it, you could create your own (unsupported) branch of Neo4j and modify that yourself to make things work, or pay Neo Technology for commercial Java 1.5 +Neo4j support. Cheers, Tobias On Sat, May 14, 2011 at 1:04 AM, Peter Neubauer < peter.neuba...@neotechnology.com> wrote: > Julien, > I think there are some parts in the server that require Java 1.6 liek the > javax.scripting , but I am not sure about the kernel. I suspect some of the > service loader extensions being Java 1.6. > > Anyone else having that info? > > Cheers, > > /peter neubauer > > GTalk: neubauer.peter > Skype peter.neubauer > Phone +46 704 106975 > LinkedIn http://www.linkedin.com/in/neubauer > Twitter http://twitter.com/peterneubauer > > http://www.neo4j.org - Your high performance graph database. > http://startupbootcamp.org/- Öresund - Innovation happens HERE. > http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. > > > On Fri, May 13, 2011 at 10:40 AM, Julien Serdaru > wrote: > > > Hi all, > > > > > > > > I am considering using Neo4j embedded into a web application. We have to > > support Java 1.5 since we deploy on client J2EE application servers which > > still run on 1.5. > > > > Looking at the documentation, it seems that Java 1.6 is a prerequisite. > > > > > > > > http://docs.neo4j.org/chunked/stable/deployment-requirements.html > > > > > > > > Are there any plans to maintain 1.5 compatibility? > > > > > > > > Thanks, > > > > > > > > ___ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Traversing all relationship types
There is a tentative new traversal API included in Neo4j that features this functionality: org.neo4j.kernel.Traversal.description().expand( org.neo4j.kernel.Traversal.expanderForAllTypes( Direction.OUTGOING ) ); You can read more about it here: http://wiki.neo4j.org/content/Traversal_Framework Cheers, Tobias On Wed, May 11, 2011 at 9:08 AM, Jean-Pierre Bergamin wrote: > Hello neo4j users > > I'm just diving into neo4j and playing around with graph algorithms and > traversers. > As a start, I just wanted to traverse the whole graph with all relationship > types in the OUTGOING direction. The traverse() method always expects a > RelationshipType. > Is there a simpler way to traverse all RelationshipTypes then building up > an > array that contains all types and the outgoing direction (as described here > http://sujitpal.blogspot.com/2009/06/custom-traverser-for-neo4j.html)? > > > Thanks for the help, > James > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Relationship iterator doesn't implement remove()
As you know it is perfectly valid for an iterator to not implement the remove() method, the fact of that being a poorly designed API is a separate issue. The main reasons for the remove() method not being implemented are: 1. It is very rarely used. Invoking the delete() method on the relationship object is easy enough anyhow. 2. It adds complexity. In order to support remove() you need to keep track of the previously returned item to know what to remove, this adds more state to the iterator and makes the code a bit more complex, which of course means a higher likelihood of bugs. The cost of this seems a bit high for supporting a method that is hardly ever used. Those are the reasons. If you have arguments that counters or invalidates those reasons please put them forward. Cheers, Tobias On Thu, May 5, 2011 at 5:12 PM, Balazs E. Pataki wrote: > Hi, > > Anothor bug like feature I just found. :-) > > Node#getRelationships() return an Iterablle, from which I can get an > Iterator. This Iterator (actually IntArrayIterator) however doesn't > implement the full Iterator interface as it throws an > UnsupportedOperationException() when remove() is called on it. (APi > documentationd doesn't mention this restriction) > > Is there any reason why this Iterator cannot implement remove()? > > I now have a wrapper around it, where I call delete() on the last > Relationship returned by next(). > > Regards, > --- > balazs > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Retrieving a node linked to the reference node takes forever
Could you provide a stack trace of what the stuck thread is doing? If you issue 'kill -3' to the process it will print a threaddump to stdout, that will contain a stack trace for the stuck thread. If you send that to me it would greatly help me diagnose the issue. Cheers, Tobias On Fri, May 6, 2011 at 12:53 PM, Pablo Pareja wrote: > Hi ! > > I just run into a problem I don't understand at all. > The scenario is actually quite simple, basically it consists of a simple > node retrieving through a relationship: > > 1. - There is a node (Protein self-interactions node) which is linked to > the > reference node by means of one > relationship (Protein self-interactions relationship). This is the > piece of code for the node creation/linkage > > public static ProteinSelfInteractionsRel proteinSelfInteractionsRel = > new ProteinSelfInteractionsRel(null); >... > > > proteinSelfInteractionsProperties.put(ProteinSelfInteractionsNode.NODE_TYPE_PROPERTY, > ProteinSelfInteractionsNode.NODE_TYPE); >long proteinSelfInteractionsNodeId = > inserter.createNode(proteinSelfInteractionsProperties); >inserter.createRelationship(inserter.getReferenceNode(), > proteinSelfInteractionsNodeId, >proteinSelfInteractionsRel, null); > > You can check the real code for this file > here< > https://github.com/pablopareja/Bio4j/blob/master/src/com/era7/bioinfo/bio4j/programs/InitBio4jDB.java > >. > (line 157) > > 2.- Later on when I try to get the id of that node: > > Iterable iterable = > manager.getReferenceNode().getRelationships(new > ProteinSelfInteractionsRel(null),Direction.OUTGOING); > logger.log(Level.INFO,"getting node"); > long proteinSelfInteractionsNodeId = > iterable.iterator().next().getEndNode().getId(); > logger.log(Level.INFO,"done!"); > > (You can also check the code for this here > < > https://github.com/pablopareja/Bio4j/blob/master/src/com/era7/bioinfo/bio4j/programs/ImportProteinInteractions.java > >(line > 117) > > It just gets stuck forever after "getting node" > > There's no exception thrown and the program just seems to freeze forever > (I've already been waiting for more than 1h with no success) > > Do you have any ideas of what could be going on?? > > I really have no clue of what could I be doing wrong... > In case it may be useful, the size of the DB at this point is around > 46.000.000 nodes and 380.000.000 relationships (only a dozen or so > coming/going to > the reference node) > > Thanks in advance, > > > Pablo > > -- > Pablo Pareja Tobes > LinkedInhttp://www.linkedin.com/in/pabloparejatobes > Twitter http://www.twitter.com/pablopareja > > http://about.me/pablopareja > http://www.ohnosequences.com > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] First-class "type" property on relationships but not nodes; why?
The RelationshipType isn't a type. It is a navigational feature. I've slapped this link around for a few years now, every time this question has been brought up: http://lists.neo4j.org/pipermail/user/2008-October/000848.html The fact that RelationshipType is a navigational feature and not a type means that there is in fact already a corresponding thing for nodes: the Indexes. But I agree that there would be things we could gain by adding a Type concept to Nodes. Such as for example better automatic indexing. But I don't know what it would look like. And I want it to be clear that such a feature is very different from what RelationshipType is today. Cheers, Tobias On Thu, May 5, 2011 at 10:29 AM, Aseem Kishore wrote: > I've found it interesting that Neo4j has a mandatory "type" property on > relationships, but not nodes. Just curious, what's the reasoning behind the > design having this distinction? > > If you say "you need to know what type of relationship these two nodes > have", I would reply, "don't you also need to know what type of nodes they > are, as well?" > > Similarly, if you say "because there can be many different types of > relationships", I would reply, "there can also be many different types of > nodes, and in both cases, there doesn't need to be". > > A perfect example is in the documentation/tutorial: movies and actors. Just > the fact that we talk about the nodes in the database as "movies" and > "actors" -- wouldn't it be helpful for the database to support that > categorization first-class? > > To be precise, it's easy for us to add a "type" property to nodes ourselves > (we do in our usage), but it's not a first-class property like > relationships, where queries and traversals can easily and naturally > specify > the type or types they expect. > > Thanks! > > Aseem > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] No applicable backend found error in Neo4j.py
If you pass the parameter log=True, you'll get more details on why it fails,
like so:
graphdb = neo4j.GraphDatabase('some/path',log=True)
That will help me figure out what the problem is in your case.
I know that the process with the current version of neo4j.py is painful.
It is mainly due to it being old. There has not been any updates in a really
long time.
I am working on a new version, where the code is simpler, which will
hopefully make people more willing to contribute. But that task keeps
getting interrupted by other things.
-tobias
On Wed, May 4, 2011 at 4:02 PM, Milena Araujo wrote:
> So I'm starting with this graph database thing and decided to give Neo4j.py
> a try, but it's being painful just to set it up.
>
> After setting all the dependencies (I'm using Jpype/Cpython) I got this
> error when I try to create a database:
>
> File "", line 1, in
> > File
> >
> "/usr/local/lib/python2.6/dist-packages/Neo4j.py-0.1_SNAPSHOT-py2.6.egg/neo4j/__init__.py",
> > line 522, in __new__
> > neo = core.load_neo(resource_uri, params)
> > File
> >
> "/usr/local/lib/python2.6/dist-packages/Neo4j.py-0.1_SNAPSHOT-py2.6.egg/neo4j/_core.py",
> > line 180, in load_neo
> > backend.initialize(**parameters)
> > File
> >
> "/usr/local/lib/python2.6/dist-packages/Neo4j.py-0.1_SNAPSHOT-py2.6.egg/neo4j/_backend/__init__.py",
> > line 61, in initialize
> > raise ImportError("No applicable backend found.")
> > ImportError: No applicable backend found.
> >
>
>
> The command that I'm using is :
>
> graphdb = neo4j.GraphDatabase('some/path')
> >
>
> I can import neo4j and pype in the command line with no problems ...
> Any ideas what it might be ?
>
> Thanks in advance :D
>
> Milena.
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Regarding Weighted Node
Hi Pooja, You could add the weight as a property on the nodes. For inserting the nodes in a particular order, if you have a complete dataset when you insert you can always sort it before you insert it. Or did I misunderstand your question? Cheers, Tobias On Tue, May 3, 2011 at 7:24 PM, pooja naik wrote: > Hi > > Is there a way in neo 4j to add weights to the node ? > > I want to add weights to the node and want to insert and delete nodes in > the increasing order of weight? > > For example: > > The following is the graph/tree representation I am interested in. > > Parent > node 1 - weight 1 > > Has children with weights > > Node 2 - weight 3, node 3- weight 5 > > > > > > > So the insertion of node4 -weight 4 will go before node3- weight 5 and > after node 2- 3 > > > > Regards > Pooja > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo Memory Tuning - Autotuning
Right. Thanks for pointing out that there is a faulty documentation in the wiki. I'll add it to the backlog of documentation fixes. Have a great weekend. -tobias On Fri, Apr 29, 2011 at 5:52 PM, Rick Bullotta wrote: > This was a case of RTFM, or rather RTWM (read the wrong manual) - the old > Neo4J Wiki stuff on Configuration Settings and Performance still comes up > first on search results, thus my confused questions. > > > -Original Message- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > On Behalf Of Tobias Ivarsson > Sent: Friday, April 29, 2011 11:46 AM > To: Neo4j user discussions > Subject: Re: [Neo4j] Neo Memory Tuning - Autotuning > > Auto tuning is what happens when you don't specify any memory mapping > parameters. > Default values was a long time ago. > > -tobias > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo Memory Tuning - Autotuning
Auto tuning is what happens when you don't specify any memory mapping parameters. Default values was a long time ago. -tobias On Fri, Apr 29, 2011 at 5:41 PM, Rick Bullotta wrote: > Cool! Didn't even know it was in there! I thought it used the default > values unless told otherwise. How can auto tuning be enabled? > > > -Original Message- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > On Behalf Of Tobias Ivarsson > Sent: Friday, April 29, 2011 11:40 AM > To: Neo4j user discussions > Subject: Re: [Neo4j] Neo Memory Tuning - Autotuning > > This is roughly what the built in auto tuning does. > In what ways is that insufficient for your needs? > > Cheers, > Tobias > > On Fri, Apr 29, 2011 at 3:43 PM, Rick Bullotta > wrote: > > > Is there any suggested way to auto-tune the various Neo memory mapping > > settings? We have such a wide variety of database scenarios and system > > configurations, that it is not possible for us to have a static set of > > configuration parameters, but we'd like to avoid requiring "human > > intervention". > > > > Curious as to if others have faced this challenge and how they solved it. > > I was thinking about asking the JVM how much memory was allocated (using > > the Runtime class) and checking the file sizes of the various Neo stores > to > > prepare a HashMap with the appropriate mapped memory settings prior to > > starting the EmbeddedGraphDatabase. I was going to allocate memory > mapping > > based on proportional sizes of the node and relationship db files. > > > > I have to think this would be generally useful capability, so I'm > wondering > > if anyone has already implemented something along those lines. > > > > Thanks, > > > > Rick > > > > ___ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > -- > Tobias Ivarsson > Hacker, Neo Technology > www.neotechnology.com > Cellphone: +46 706 534857 > _______ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo Memory Tuning - Autotuning
This is roughly what the built in auto tuning does. In what ways is that insufficient for your needs? Cheers, Tobias On Fri, Apr 29, 2011 at 3:43 PM, Rick Bullotta wrote: > Is there any suggested way to auto-tune the various Neo memory mapping > settings? We have such a wide variety of database scenarios and system > configurations, that it is not possible for us to have a static set of > configuration parameters, but we'd like to avoid requiring "human > intervention". > > Curious as to if others have faced this challenge and how they solved it. > I was thinking about asking the JVM how much memory was allocated (using > the Runtime class) and checking the file sizes of the various Neo stores to > prepare a HashMap with the appropriate mapped memory settings prior to > starting the EmbeddedGraphDatabase. I was going to allocate memory mapping > based on proportional sizes of the node and relationship db files. > > I have to think this would be generally useful capability, so I'm wondering > if anyone has already implemented something along those lines. > > Thanks, > > Rick > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j.py + django relationship to same node type
Pythons evaluation model prevents you from referencing things that have not
been defined yet.
In this case you are trying to reference Person before the class definition
of Person is done.
Just like in regular Django models this can be worked around by just
providing the name of the class as a string. Then the actual class will be
looked up at a later time.
Cheers,
tobias
On Thu, Apr 28, 2011 at 10:31 PM, Hammad wrote:
> hey it my first try at using django + neo4j together. and i have already
> run
> into a problem:
>
> here is my model:
>
> from neo4j.model import django_model as neoModel
>
> class Person(neoModel.NodeModel):
>name = neoModel.Property(indexed=True)
>age = neoModel.Property()
>related = neoModel.Relationship(Person, type=neoModel.Outgoing.related)
>href = property(lambda self: ('/idea/%s/' % (self.node.id,)))
>
>def __unicode__(self):
>return u'"%s"' % (self.name)
>
> while testing it by starting the shell from manage.py i get this error when
> i import the models(from myapp import models):
>
> Traceback (most recent call last):
> File "", line 1, in
> File "/home/hammad/Dev/Project/Myproj/myapp/models.py", line 4, in
>
>class Person(neoModel.NodeModel):
> File "/home/hammad/Dev/Project/Myproj/myapp/models.py", line 7, in Person
>related = neoModel.Relationship(Person, type=neoModel.Outgoing.related)
> NameError: name 'Person' is not defined
>
> any ideas on how i can create a relationship between two nodes of the same
> time. Its such a basic use-case that i am sure i am missing something...
>
> --
> Hammad Ullah
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How Do You Configure neo4j Not to Reuse IDs?
No, that patch has not made it into the mainline. Why do you want to disable id reuse? -tobias On Thu, Apr 28, 2011 at 3:10 PM, espeed wrote: > I found the mentions of a patch from about a year ago... > > http://lists.neo4j.org/pipermail/user/2010-June/003964.html > > Has something like this with a configuration param made it into the main > branch? > > Thanks. > > - James > > -- > View this message in context: > http://neo4j-user-list.438527.n3.nabble.com/How-Do-You-Configure-neo4j-Not-to-Reuse-IDs-tp2874501p2874642.html > Sent from the Neo4J User List mailing list archive at Nabble.com. > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] ClosedChannelExceptions in highly concurrent environment
Hi Jennifer,
I'd first like to thank you for the testing and analysis you've done. Very
useful stuff. Do you think you could send some test code our way that
reproduces this issue?
This is actually the first time this issue has been reported, so I wouldn't
say it is a common issue. My guess is that your thread volume triggered a
rare condition that wouldn't be encountered otherwise.
I'm also curious to know how you found the source of the interruptions. When
I debug thread interruptions I've never been able to find out where the
thread got interrupted from without doing tedious procedures of breakpoint +
logging + trying to match thread ids. If you have a better method for doing
that I'd very much like to know.
I think we should focus the effort on fixing the interruption issue if we
can. And I believe we would be able to do that if the interruptions do in
fact originate from where you say they do. But the suggestion of being able
to switch the lucene directory implementation is still interesting, but as
you point out since it has issues on some platforms it would be better if we
could be rid of the interruption issue.
Cheers,
Tobias
On Thu, Apr 28, 2011 at 12:41 AM, Jennifer Hickey wrote:
> Hello,
> I've been running some tests w/approx 400 threads reading various indexed
> property values. I'm running on 64 bit Linux. I was frequently seeing the
> ClosedChannelException below. The javadoc on Lucene's NIOFSDirectory states
> that "Accessing this class either directly or indirectly from a thread while
> it's interrupted can close the underlying file descriptor immediately if at
> the same time the thread is blocked on IO. The file descriptor will remain
> closed and subsequent access to {@link NIOFSDirectory} will throw a {@link
> ClosedChannelException}. If your application uses either {@link
> Thread#interrupt()} or {@link Future#cancel(boolean)} you should use {@link
> SimpleFSDirectory} in favor of {@link NIOFSDirectory}."
>
> A bit of debugging revealed that the Thread.interrupts were coming from
> Neo4j, specifically in RWLock and MappedPersistenceWindow. So it seems like
> this would be a common problem, though perhaps I am missing something?
>
> SimpleFSDirectory seems a bit of a performance bottleneck, so I switched to
> MMapDirectory and the problem did go away. I didn't see a way to switch
> implementations w/out modifying neo4j code, so I changed LuceneDataSource as
> follows:
>
> static Directory getDirectory( String storeDir,
>IndexIdentifier identifier ) throws IOException
> {
>MMapDirectory dir=new MMapDirectory(getFileDirectory( storeDir,
> identifier), null);
>if(MMapDirectory.UNMAP_SUPPORTED) {
>dir.setUseUnmap(true);
>}
>return dir;
> }
>
> So I'm wondering if others have seen this problem and/or if there is a
> recommended solution? Our product runs on quite a few different operating
> systems, so I have some reservations about using MMapDirectory as well
> (javadoc speaks of a few caveats on Windows, 64 vs 32, etc). Also, I'd
> rather not maintain a patched version of the neo4j code if avoidable.
>
> Thanks!
> Jennifer
>
> Exception:
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88)
> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:613)
> at
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:161)
> at
> org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:139)
> at
> org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:285)
> at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:160)
> at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:86)
> at
> org.apache.lucene.index.codecs.DeltaBytesReader.read(DeltaBytesReader.java:40)
> at
> org.apache.lucene.index.codecs.PrefixCodedTermsReader$FieldReader$SegmentTermsEnum.next(PrefixCodedTermsReader.java:469)
> at
> org.apache.lucene.index.codecs.PrefixCodedTermsReader$FieldReader$SegmentTermsEnum.seek(PrefixCodedTermsReader.java:385)
> at org.apache.lucene.index.TermsEnum.seek(TermsEnum.java:68)
> at org.apache.lucene.index.Terms.docFreq(Terms.java:53)
> at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:898)
> at org.apache.lucene.index.IndexReader.docFreq(IndexReader.java:882)
> at
> org.apache.lucene.index.DirectoryReader.docFreq(DirectoryReader.java:687)
>
>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Question about REST interface concurrency
The PresistenceWindows are flushed only when the transaction log is rotated. Writes to the transaction log is flushed when the transaction commits though. If you wish to look at the code for this, have a look at for instance: https://github.com/neo4j/community/blob/master/kernel/src/main/java/org/neo4j/kernel/impl/transaction/xaframework/XaLogicalLog.java Cheers, Tobias On Wed, Apr 27, 2011 at 4:31 AM, Stephen Roos wrote: > Hi Jim, > > From what I understand, it flushes with various granularities, though I'd > suspect that it's not flushing after writes the size of empty nodes, so this > is certainly a possible bottleneck point. I've been looking through the > code and don't see exactly where the flush takes place. Can you point me at > the right class? > > I did come across the PersistenceWindowPool class which seems to come into > play when the underlying node record is updated during the transaction > commit. It looks as if the windows are mapped over contiguous blocks of the > primitives ID space and that because the new node IDs are typically > sequential, each of my create-node operations is likely to target the same > window. These windows are locked and waiting threads are queued up to wait > for the locking thread to notify on unlock. Am I reading the code > correctly? If so, do you have any thoughts on how we might remove that > bottleneck? > > Thanks again for your help, > Stephen > > > -Original Message- > From: Mattias Persson [mailto:matt...@neotechnology.com] > Sent: Tuesday, April 26, 2011 12:19 AM > To: Neo4j user discussions > Subject: Re: [Neo4j] Question about REST interface concurrency > > Does your disk benchmark tests flush the data to disk or just write to > it, making file system / OS flush when ever it feel like it (making it > much faster, of course)? > > 2011/4/25 Stephen Roos : > > Hi Jim, > > > > I took a look at my disk utilization and I'm only getting up to about > 9379 KBps (write). My disk benchmarking tests show max write rates to be > around 220 MBps, so I shouldn't be maxed out there. Interestingly, I don't > see that much data in the graph.db directory (I see about 15 MB there after > creating 150k empty nodes, no relationships, no index). The largest file is > nioneo_logical.log.1 (14 MB), the next largest is the neostore.nodestore.db > (1.3 MB). I don't know if that information is helpful, but I thought it was > a bit strange that I'm sustaining disk write rates of > 9 MBps for over 40 > secs yet I don't have anywhere close to 9 * 40 MB of data. > > > > I do wonder about the flush operation though. Flush is a blocking > operation, maybe that's the bottleneck even though the disk isn't over > utilized. I'll look into that. Let me know if you have any other ideas. > > > > Thanks! > > Stephen > > > > -Original Message- > > From: Jim Webber [mailto:j...@neotechnology.com] > > Sent: Friday, April 22, 2011 3:34 AM > > To: Neo4j user discussions > > Subject: Re: [Neo4j] Question about REST interface concurrency > > > > Hi Stephen, > > > > I think the network IO you've measured is consistent with the rest of the > behaviour your've described. > > > > What I'm thinking is that you're simply reaching the limits of create > transaction->create a node->complete transaction->flush to filesystem (that > is, you're basically testing disk write speed/seek time/etc). > > > > Can you check how busy your IO to disk is? I expect it'll be relatively > high. > > > > Jim > > ___ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > -- > Mattias Persson, [matt...@neotechnology.com] > Hacker, Neo Technology > www.neotechnology.com > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4j on OSGi
t;> for example. Those features could also for example look for exposed > > >>> GraphDatabase and then register management of webadmin features > > >>> automatically. > > >>> I don't know if it would be useful or even if that makes sense > > regarding > > >>> neo4j use cases. Let me know what you think about that. > > >>> > > >>> Nicolas. > > >>> ___ > > >>> Neo4j mailing list > > >>> User@lists.neo4j.org > > >>> https://lists.neo4j.org/mailman/listinfo/user > > >>> > > >> ___ > > >> Neo4j mailing list > > >> User@lists.neo4j.org > > >> https://lists.neo4j.org/mailman/listinfo/user > > > > > > ___ > > > Neo4j mailing list > > > User@lists.neo4j.org > > > https://lists.neo4j.org/mailman/listinfo/user > > > > ___ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] REST results pagination
On Thu, Apr 21, 2011 at 11:18 PM, Michael Hunger <
michael.hun...@neotechnology.com> wrote:
> Rick,
>
> great thoughts.
>
> Good catch, forgot to add the in-graph representation of the results to my
> mail, thanks for adding that part. Temporary (transient) nodes and
> relationships would really rock here, with the advantage that with HA you
> have them distributed to all cluster nodes.
> Certainly Craig has to add some interesting things to this, as those
> resemble probably his in graph indexes / R-Trees.
>
> As traversers are lazy a count operation is not so easily possible, you
> could run the traversal and discard the results. But then the client could
> also just pull those results until it reaches its
> internal tresholds and then decide to use more filtering or stop the
> pulling and ask the user for more filtering (you can always retrieve n+1 and
> show the user that there are more that "n" results available).
>
> The index result size() method only returns an estimate of the result size
> (which might not contain currently changed index entries).
>
> Please don't forget that a count() query in a RDBMS can be as ridicully
> expensive as the original query (especially if just the column selection was
> replaced with count, and sorting, grouping etc was still left in place
> together with lots of joins).
>
> Sorting on your own instead of letting the db do that mostly harms the
> performance as it requires you to build up all the data in memory, sort it
> and then use it. Instead of having the db do that more efficiently, stream
> the data and you can use it directly from the stream.
>
throw new SlapOnTheFingersException("sometimes the application developer can
do a better job since she has better knowledge of the data, the database
only has generic knowledge");
Since Jake had already mentioned (in this very thread) that he expected one
of those, I thought I might as well throw one in there.
I agree with the analysis of count(), as the name ("count") implies, it will
have to run the entire query in order to count the number of resulting
items.
About sorting I'm torn. The perception of sorting in the database being slow
that Rick points to is one that I've seen a lot. When you hand the
responsibility of sorting to the database you hide the fact that sorting is
an expensive operation, it does require reading in all data in order to sort
it. People often expect databases to be "smarter" than that, since they
sometimes are, but that is pretty much only when reading straight from an
index and not doing much more. A generic sort of data can never be better
than O(log(n!)) [O(log(n!)) is almost equal to, and commonly rounded to the
easier to compute function O(n log(n))]. If you put the responsibility of
sorting in the hands of the application you can sometimes utilize knowledge
about the data to do a more efficient sorting than the database could have
done. Most often by simply doing an application level filtering of the data
before sorting it, based on some filtering that could not be transfered to
the database query. This does however make the work of the application
developer slightly more tedious, which is why I think it would be sensible
to have support for sorting on the database level, and hope that users will
be sensible about using it, and not assume magic from it.
Something I find very interesting is the concept of semi-sorted data.
Semi-sorted data is often good enough, easier to achieve, and quite easy to
then sort completely if that is required. Examples of semi-sorted data could
be data in an order that satisfies the heap property. Or for spatial queries
returning the closest hits first, but not necessarily in perfect order, say
returning the hits within a miles radius first, before the ones in a radius
between 1-10 miles, and so on, without requiring the hits in each 'segment'
to be perfectly ordered by distance. Breadth first order is another example
of semi-sorted data, that could be used when traversing data as you've
outlined with "paging nodes", or similarly "grouped by parent node"-order.
I must say that I really enjoy following this discussion. I really like the
idea of streaming, since I think that can be implemented more easily than
paging, while satisfying many of the desired use cases. But I still want to
hear more arguments for and against both alternatives. And as has already
been pointed out, they aren't mutually exclusive.
I'll keep listening in on the conversation, but I don't have much more to
add at this point. I have one desire for the structure of the conversation
though. When you quote what someone else has said before you, could you
please include who that person was, it makes going back and reading the full
context easier.
Cheers,
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Strange performance difference on different machines
Sorry I got a bit distracted when writing this. I should have added that I then want you to send the results of running that benchmark to me so that I can further analyze what the cause of these slow writes might be. Thank you, Tobias On Wed, Apr 20, 2011 at 12:26 PM, Tobias Ivarsson < tobias.ivars...@neotechnology.com> wrote: > I agree that 16 transactions / second is much slower than what I usually > see on linux, even with a slow file system configuration. But I still > believe this is either disk or filesystem being slow. Could you please go > through the file system benchmarking outlined on this wiki page: > http://wiki.neo4j.org/content/Linux_Performance_Guide > > Cheers, > Tobias > > > On Tue, Apr 19, 2011 at 10:11 PM, Bob Hutchison > wrote: > >> Hi Tobias, >> >> On 2011-04-19, at 1:48 AM, Tobias Ivarsson wrote: >> >> > Hi Bob, >> > >> > What happens here is that you perform a tiny operation in each >> transaction, >> > so what you are really testing here is how fast your file system can >> flush, >> > because with such tiny transactions all of the time is going to be spent >> in >> > transactional overhead (i.e. flushing transaction logs to the disk). >> > >> > The reason you see such large differences between Mac OS X and Linux is >> > because Mac OS X cheats. Flushing a file (fdatasync) on Mac does pretty >> much >> > nothing. The only thing Mac OS X guarantees is that it will write the >> data >> > that you just flushed before it writes the next data block you flush, so >> > called "ordered writes". This means that you could potentially get >> data-loss >> > on hard failure, but never in a way that makes your data internally >> > inconsistent. >> >> Okay, that's makes some sense. Thanks for the information. >> >> > >> > So to give a short answer to your questions: >> > 1) The linux number is reasonable, Mac OS X cheats. >> > 2) What you are testing is the write speed of your disk for writing >> small >> > chunks of data. >> >> So you're thinking that 16 or 17 writes is what should be expected? >> >> Cheers, >> Bob >> >> > >> > Cheers, >> > Tobias >> > >> > On Mon, Apr 18, 2011 at 10:57 PM, Bob Hutchison < >> hutch-li...@recursive.ca>wrote: >> > >> >> Hi, >> >> >> >> Using Neo4j 1.3 and the Borneo (Clojure) wrapper I'm getting radically >> >> different performance numbers with identical test code. >> >> >> >> The test is a simple-minded: create two nodes and a relation between >> them. >> >> No properties, no indexes, all nodes and relations are different. >> >> >> >> On OS X, it takes about 50s to perform that operation 50,000 times, < >> 0.8s >> >> to do it 500 times. It uses roughly 30-40% of one core to do this. >> >> >> >> On linux it takes about 30s to perform that operation 500 times. The >> CPU >> >> usage is negligible (really negligible... almost none). >> >> >> >> I cannot explain the difference in behaviour. >> >> >> >> I have two questions: >> >> >> >> 1) is either of these a reasonable number? I hoping the OS X numbers >> are >> >> not too fast. >> >> >> >> 2) any ideas as to what might be the cause of this? >> >> >> >> The Computers are comparable. The OS X is a 2.8 GHz i7, the linux box >> is a >> >> 3.something GHz Xeon (I don't remember the details). >> >> >> >> Thanks in advance for any help, >> >> Bob >> >> >> >> _______ >> >> Neo4j mailing list >> >> User@lists.neo4j.org >> >> https://lists.neo4j.org/mailman/listinfo/user >> >> >> > >> > >> > >> > -- >> > Tobias Ivarsson >> > Hacker, Neo Technology >> > www.neotechnology.com >> > Cellphone: +46 706 534857 >> > ___ >> > Neo4j mailing list >> > User@lists.neo4j.org >> > https://lists.neo4j.org/mailman/listinfo/user >> >> >> Bob Hutchison >> Recursive Design Inc. >> http://www.recursive.ca/ >> weblog: http://xampl.com/so >> >> >> >> >> ___ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> > > > > -- > Tobias Ivarsson > Hacker, Neo Technology > www.neotechnology.com > Cellphone: +46 706 534857 > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Strange performance difference on different machines
I agree that 16 transactions / second is much slower than what I usually see on linux, even with a slow file system configuration. But I still believe this is either disk or filesystem being slow. Could you please go through the file system benchmarking outlined on this wiki page: http://wiki.neo4j.org/content/Linux_Performance_Guide Cheers, Tobias On Tue, Apr 19, 2011 at 10:11 PM, Bob Hutchison wrote: > Hi Tobias, > > On 2011-04-19, at 1:48 AM, Tobias Ivarsson wrote: > > > Hi Bob, > > > > What happens here is that you perform a tiny operation in each > transaction, > > so what you are really testing here is how fast your file system can > flush, > > because with such tiny transactions all of the time is going to be spent > in > > transactional overhead (i.e. flushing transaction logs to the disk). > > > > The reason you see such large differences between Mac OS X and Linux is > > because Mac OS X cheats. Flushing a file (fdatasync) on Mac does pretty > much > > nothing. The only thing Mac OS X guarantees is that it will write the > data > > that you just flushed before it writes the next data block you flush, so > > called "ordered writes". This means that you could potentially get > data-loss > > on hard failure, but never in a way that makes your data internally > > inconsistent. > > Okay, that's makes some sense. Thanks for the information. > > > > > So to give a short answer to your questions: > > 1) The linux number is reasonable, Mac OS X cheats. > > 2) What you are testing is the write speed of your disk for writing small > > chunks of data. > > So you're thinking that 16 or 17 writes is what should be expected? > > Cheers, > Bob > > > > > Cheers, > > Tobias > > > > On Mon, Apr 18, 2011 at 10:57 PM, Bob Hutchison < > hutch-li...@recursive.ca>wrote: > > > >> Hi, > >> > >> Using Neo4j 1.3 and the Borneo (Clojure) wrapper I'm getting radically > >> different performance numbers with identical test code. > >> > >> The test is a simple-minded: create two nodes and a relation between > them. > >> No properties, no indexes, all nodes and relations are different. > >> > >> On OS X, it takes about 50s to perform that operation 50,000 times, < > 0.8s > >> to do it 500 times. It uses roughly 30-40% of one core to do this. > >> > >> On linux it takes about 30s to perform that operation 500 times. The CPU > >> usage is negligible (really negligible... almost none). > >> > >> I cannot explain the difference in behaviour. > >> > >> I have two questions: > >> > >> 1) is either of these a reasonable number? I hoping the OS X numbers are > >> not too fast. > >> > >> 2) any ideas as to what might be the cause of this? > >> > >> The Computers are comparable. The OS X is a 2.8 GHz i7, the linux box is > a > >> 3.something GHz Xeon (I don't remember the details). > >> > >> Thanks in advance for any help, > >> Bob > >> > >> ___ > >> Neo4j mailing list > >> User@lists.neo4j.org > >> https://lists.neo4j.org/mailman/listinfo/user > >> > > > > > > > > -- > > Tobias Ivarsson > > Hacker, Neo Technology > > www.neotechnology.com > > Cellphone: +46 706 534857 > > ___ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > > Bob Hutchison > Recursive Design Inc. > http://www.recursive.ca/ > weblog: http://xampl.com/so > > > > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Strange performance difference on different machines
Hi Bob, What happens here is that you perform a tiny operation in each transaction, so what you are really testing here is how fast your file system can flush, because with such tiny transactions all of the time is going to be spent in transactional overhead (i.e. flushing transaction logs to the disk). The reason you see such large differences between Mac OS X and Linux is because Mac OS X cheats. Flushing a file (fdatasync) on Mac does pretty much nothing. The only thing Mac OS X guarantees is that it will write the data that you just flushed before it writes the next data block you flush, so called "ordered writes". This means that you could potentially get data-loss on hard failure, but never in a way that makes your data internally inconsistent. So to give a short answer to your questions: 1) The linux number is reasonable, Mac OS X cheats. 2) What you are testing is the write speed of your disk for writing small chunks of data. Cheers, Tobias On Mon, Apr 18, 2011 at 10:57 PM, Bob Hutchison wrote: > Hi, > > Using Neo4j 1.3 and the Borneo (Clojure) wrapper I'm getting radically > different performance numbers with identical test code. > > The test is a simple-minded: create two nodes and a relation between them. > No properties, no indexes, all nodes and relations are different. > > On OS X, it takes about 50s to perform that operation 50,000 times, < 0.8s > to do it 500 times. It uses roughly 30-40% of one core to do this. > > On linux it takes about 30s to perform that operation 500 times. The CPU > usage is negligible (really negligible... almost none). > > I cannot explain the difference in behaviour. > > I have two questions: > > 1) is either of these a reasonable number? I hoping the OS X numbers are > not too fast. > > 2) any ideas as to what might be the cause of this? > > The Computers are comparable. The OS X is a 2.8 GHz i7, the linux box is a > 3.something GHz Xeon (I don't remember the details). > > Thanks in advance for any help, > Bob > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Multitenancy in neo4j
If you by multitenancy simply mean that you want to have multiple separate graph database, you can do that as easily as: GraphDatbaseService graphdb1 = new EmbeddedGraphDatabase( "/path/to/your/first/graphdb" ); GraphDatbaseService graphdb2 = new EmbeddedGraphDatabase( "/path/to/your/second/graphdb" ); Since that is already possible in the embedded scenario, the multitencncy support that has been discussed for the future is a server specific feature. That work for the server might bring some new capabilities to the embedded setup as well, but the main part of it is still going to be as exemplified above. Or you could of course do as Jim suggested and make the sub-graph distinction an application level separation. One thing to keep in mind when starting multiple EmbeddedGraphDatabase instances in the same JVM (and on the same computer for that matter) is that the automatic configuration will assume that it is the only graph database running on that computer, sou you will need to supply manual configuration. See: http://wiki.neo4j.org/content/Configuration_Settings Cheers, Tobias On Mon, Apr 18, 2011 at 10:21 AM, ntankovic wrote: > Hi all, > > I'm facing a situation to develop a multitenant graph environment. I heard > from some of you guys that this is planned for neo4j server. Is this neo4j > server only or will I also be possible to use this on neo4j embedded > engine? > > -- > View this message in context: > http://neo4j-user-list.438527.n3.nabble.com/Multitenancy-in-neo4j-tp2833480p2833480.html > Sent from the Neo4J User List mailing list archive at Nabble.com. > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Architecturing Neo4J database
What is the required number of relationships? I'll answer your questions inline below. On Sun, Apr 17, 2011 at 7:27 PM, Dima Gutzeit wrote: > Oops, small correction. The required number of nodes is 2 billions and not > 100 millions as originally stated. > > On Sun, Apr 17, 2011 at 8:16 PM, Dima Gutzeit > wrote: > > > Dear list members, > > > > I have a need for storage engine (graph based) for a server I am > building. > > > > Here are the requirements: > > > > 1. Support for High Availability > You can tick that box with Neo4j :) > > 2. Ability to store around 100 millions of nodes, estimated data size of > > around 5 terabytes. > Measuring data in bytes is blunt and difficult. Statistically, all users trying to do that before using Neo4j for at least a while has gotten it wrong in one way or another. Is it ok if I take this figure with a grain of salt? Your statement of "2 bilion nodes" is more helpful, add to that the estimated number of relationships (either per node or in total) and the estimated number of properties for each node and relationship and we'll have a better model for estimating both on-disk size and how to best configure your Neo4j installation for optimal cache utilization. > > > > I've been reading about different options that Neo4J provides and I need > to > > make a final decision about the layout, few questions: > > > > - I expect mostly-read-access to the database and I've been reading about > > the HA approach, by replicating the data across several nodes. Is there > an > > option to use a centralized storage server (NAS/SAN) and working with > > several nodes (for scaling) but without data replication ? > No. Since Neo4j requires exclusive file locks for the storage files to guarantee write consistency. It also doesn't work well even if you have a single write-instance and let all other instances be read-only instances used for only scaling reads. This is because Neo4j relies heavily on caching to get good read performance. If the underlying storage data is changed by the write-master the read slaves will not know about it and thus suffer from stale caches which can lead to an inconsistent view of the graph. The Neo4j HA solution provides the appropriate replication and cache invalidation to make this work consistently across multiple instances. > > > > - The documentation describes HA installation with embedded neo4j > instance, > > can the same be configured for a standalone server ? > Yes. > > > > - If using an embedded version of the database, can still the webadmin be > > used ? > Not yet. This is being considered for future versions. > > > > - What is supported request rate / throughput of a single Neo4j server, > is > > there benchmarking data I can consult with including the hardware config > of > > a testbed ? > The figures for an embedded Neo4j setup is between 1000 to 2000 relationships traversed per millisecond for fully cached data. With approximately 10% of that for data only in the low level (file system) cache. The server setup used the same graph database internally, but adds some per-request network overhead to that. I don't have any benchmark figures. Could someone fill in? Cheers, -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Taking a Snapshot.
Hi Guru, Actually Emil is not completely up to speed with the current state of the online backup feature. Our previous online backup feature worked this way, since all it did was copy the transaction logs and replay those on the backup target. This had the drawback of requiring an equal database (from a previous state - usually the initial state) to exist at the target location. It also requires all transaction logs to exist on the source machine from the time when the target baseline copy was created to the current point in time. This functionality still exists in the current backup tool as "incremental backup". What the is added though is the ability to do full backup. When you do a full backup you don't need to have all the transaction logs around at the source db. What we do then is copy the store files. But since we want the source database to be up an running as usual we don't lock the store files. This means that they can be modified while we are copying them. This in turn means that if we only copied the store files the backup target could contain a database at an inconsistent state. To avoid that we record which transaction the db was at when the backup started. When the backup of the store files is complete, we send the transaction logs of all transactions from the recorded start-transaction to the transaction the db was at when the copying finished. Playing these transactions back on the backup target ensures that the store files get to a consistent state. This means that the state of the backup target after backup will be the same as it was on the source when the backup process completed there. So for you this means that you would have to use incremental backup to achieve your goals. What I would suggest is to do that backup in two phases: 1. Do a full backup ahead of the time for which you want the snapshot from. 2. At the point at which you want a snapshot, do an incremental backup to the same backup target as the previous full backup. Since incremental backup copies the transaction logs up until the point when it started the above procedure will give you the desired outcome. Cheers, Tobias On Sat, Apr 16, 2011 at 8:51 AM, Guru GV wrote: > Wow ! Amazingly quick response, Thanks Emil! > > > On Sat, Apr 16, 2011 at 12:20 PM, Emil Eifrem >wrote: > > > On Fri, Apr 15, 2011 at 23:47, Guru GV wrote: > > > What happens if I initiate the Backup and then couple of more > > transactions > > > are performed on the DB and only after that the Backup finishes. > > > In that case - is it guaranteed that the those Txs (that started after > > the > > > backup was initiated)* are not included* and the backup that I get is > > > exactly what the *DB was at the point of time I requested for a backup > ?* > > > > Yes. It will behave as you expect in the scenario you outline. > > > > Cheers, > > > > -- > > Emil Eifrém, CEO [e...@neotechnology.com] > > Neo Technology, www.neotechnology.com > > Cell: +46 733 462 271 | US: 206 403 8808 > > http://blogs.neotechnology.com/emil > > http://twitter.com/emileifrem > > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Mac Pro Question ...
There are known issues with "neo4j install" on the mac. We are working on that. But just running should be fine, i.e. "neo4j start", I do that on my mac all the time. Roughly half of the Neo Technology devteam have Macs as their main development machines. Cheers, Tobias On Fri, Apr 15, 2011 at 6:19 AM, Rense Lange wrote: > > > > Is there anyone who is running neo4j on their Mac? How did you do it? I > have a Mac pro, and I can't seem to get it going. Right now, the > installation methods provided on the web don't work. I hope there is a way > around this. Help! > > Rense Lange > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] 20 billion primitives machine
Hi Jamie, Very interesting use case you have there. If you could just provide a few more bits of information about your data, I'd be able to provide a better analysis. * Is the distribution of relationships uniform across the nodes. If not, how much does it vary? * What kind of operation you want to do in the 50ms latency that you've specified. You just mentioned "relatively shallow traversals", approximately how deep is that? The key figure to get at is how many relationships you need to traverse, and that is a function of the depth of the traversal and the distribution of the relationships. * Is the 50ms latency for each single request? What is the estimated number of concurrent requests? * What kind of properties are those 32 byte values? ASCII strings of length 32? * The bulk loading phase you describe, is it a one time import of initial data, or a regularly recurring thing? * In the bulk load phase, what form does the data have? Would it be possible to have that data in a format where each node is uniquely identified by a number (in both the node data listing, and the edge-list). Finally, do you have test data for these sizes? If possible I'd love to work with you on this, to get an good use case to work on for improving the large data story in Neo4j. Cheers, Tobias On Thu, Apr 14, 2011 at 3:12 PM, Jamie Stephens wrote: > Folks, > > I've got an application that has (will have) about 2 billion vertexes > and maybe 8 billion edges (?). Maybe an avg of 4 properties per > vertex -- with maybe an avg of 32 bytes/value. So I guess that's 16 > billion primitives. Let's round to 20 billion. My edges estimate is > a relatively uninformed guess. Just starting to dig into the data. > > Traversals will be relatively shallow. Concurrent access. Throughput > is more important than latency. But latency should be better than > maybe 50ms 99% of the time (allowing for some cache warming and some > GC). I don't know much yet about locality. I'm not sure yet how > sensitive the app will be to long GCs. > > We will need to do a big batch load, and writes will need to be fast > in that phase. After that, we'll see more reads that writes. So I > imagine a config for the batch load and another config for production. > > I understand cache sharding, application-level partitioning, and so > forth. I'm wondering what I can do on a single machine -- and what > that machine should look like. > > http://docs.neo4j.org/chunked/stable/configuration-jvm.html and > http://wiki.neo4j.org/content/Neo4j_Performance_Guide are encouraging. > And having knobs as documented at > http://wiki.neo4j.org/content/Configuration_Settings is great. Nice > work! > > I'm hoping I might be able to get away with 128GB RAM on 12 cores with > data striped over a handful of disks (SSDs if required). We'll > probably also need a cluster for both traffic and availability, but > that's another topic. > > Does anybody have experience with a data set like this on a similar > machine? How much RAM and how much disk -- and what kinds and in what > configuration? Latency, throughput, general experience? Any > production deployments? > > I'd appreciate any guidance or feedback. I'm happy to summarize later > if that'd be helpful. > > BTW, my testbed uses Clojure with clojure.contrib.server-socket and > https://github.com/wagjo/borneo. Very convenient! > > --Jamie > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo Server slow on frequent requests
That one looks much more interesting. I'm too tired right now, but I'll look at it tomorrow. On Wed, Apr 13, 2011 at 11:50 AM, Dario Rexin wrote: > Hi Tobias, > > I was already sending requests to the server in the last dump. Here is > another, hopefully this one is more helpful. The longer I request data, the > longer it takes for the server to answer. After some time it frequently > freezes for up to several seconds without answering to any of the requests. > > https://gist.github.com/917283 > > Cheers, > > Dario > > Am 13.04.11 11:14 schrieb "Tobias Ivarsson" unter > : > > > Hi Dario, > > > > This dump looks perfectly fine, the expected threads are there, but they > are > > all idle waiting for work. > > > > When I asked for a thread dump, I wanted one from when the server was > under > > load and you experienced problems. Sorry for not being clear about that. > > > > Cheers, > > Tobias > > > > On Wed, Apr 13, 2011 at 10:34 AM, Dario Rexin > wrote: > > > >> Hey, > >> > >> Somehow my attached files always get deleted. Heres the dump: > >> > >> https://gist.github.com/917199 > >> > >> > >> Cheers, > >> > >> Dario > >> > >> Am 13.04.11 10:30 schrieb "Dario Rexin" unter : > >> > >>> Hi Tobias, > >>> > >>> Here's the thread dump you asked for. Thank you for taking a look at > >> this. > >>> > >>> > >>> Cheers, > >>> > >>> Dario > >>> > >>> > >>> Am 12.04.11 22:16 schrieb "Tobias Ivarsson" unter > >>> : > >>> > >>>> Hi Dario, > >>>> > >>>> Looking at that picture it is indeed clear that a number of threads > are > >>>> waiting for something. What is not shown is the more important > >> information > >>>> about *what* they are waiting for. I would love to get information > like > >> that > >>>> in order to investigate the cause of the performance problem you are > >> seeing. > >>>> If you could send a thread dump instead of a screenshot that would be > a > >> lot > >>>> more useful, since that would contain information about contention > that > >> I > >>>> could actually analyze. The easiest way to get a thread dump is by > >> sending > >>>> the SIGQUIT signal (kill -3) to the JVM process running Neo4j. > >>>> > >>>> Cheers, > >>>> Tobias > >>>> > >>>> On Tue, Apr 12, 2011 at 6:35 PM, Dario Rexin > >> wrote: > >>>> > >>>>> Hi all, > >>>>> > >>>>> Due to huge performance issues with some of our neo queries I > profiled > >> my > >>>>> calls on the neo server. The profiling shows, that up to 85% of the > >> time the > >>>>> threads are waiting for other threads. I don¹t understand what¹s > going > >> on > >>>>> there. Hopefully someone with a deeper knowledge can help me. Am I > >> doing > >>>>> something wrong, or is it normal, that most of the time the threads > are > >>>>> blocking each other? Her is a screenshot, showing the results of my > >>>>> profiling: http://i.imgur.com/eIfam.jpg > >>>>> > >>>>> > >>>>> Thanks in advice, > >>>>> > >>>>> Dario > >>>>> ___ > >>>>> Neo4j mailing list > >>>>> User@lists.neo4j.org > >>>>> https://lists.neo4j.org/mailman/listinfo/user > >>>>> > >>>> > >>>> > >> > >> -- > >> Dario Rexin > >> Junior Manager Engineering > >> dario.re...@xing.com > >> > >> XING AG > >> Gaensemarkt 43, 20354 Hamburg, Germany > >> > >> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807 > >> Exec. Board (Vorstand): Dr. Stefan Groß-Selbeck (Vorsitzender), Ingo > Chu, > >> Dr. Helmut Becker, Jens Pape > >> Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Dr. Neil > >> Sunderland > >> > >> Please join my network on XING: > >> https://www.xing.com/profile/Dario_Rexin >
Re: [Neo4j] Neo Server slow on frequent requests
Hi Dario, This dump looks perfectly fine, the expected threads are there, but they are all idle waiting for work. When I asked for a thread dump, I wanted one from when the server was under load and you experienced problems. Sorry for not being clear about that. Cheers, Tobias On Wed, Apr 13, 2011 at 10:34 AM, Dario Rexin wrote: > Hey, > > Somehow my attached files always get deleted. Heres the dump: > > https://gist.github.com/917199 > > > Cheers, > > Dario > > Am 13.04.11 10:30 schrieb "Dario Rexin" unter : > > > Hi Tobias, > > > > Here's the thread dump you asked for. Thank you for taking a look at > this. > > > > > > Cheers, > > > > Dario > > > > > > Am 12.04.11 22:16 schrieb "Tobias Ivarsson" unter > > : > > > >> Hi Dario, > >> > >> Looking at that picture it is indeed clear that a number of threads are > >> waiting for something. What is not shown is the more important > information > >> about *what* they are waiting for. I would love to get information like > that > >> in order to investigate the cause of the performance problem you are > seeing. > >> If you could send a thread dump instead of a screenshot that would be a > lot > >> more useful, since that would contain information about contention that > I > >> could actually analyze. The easiest way to get a thread dump is by > sending > >> the SIGQUIT signal (kill -3) to the JVM process running Neo4j. > >> > >> Cheers, > >> Tobias > >> > >> On Tue, Apr 12, 2011 at 6:35 PM, Dario Rexin > wrote: > >> > >>> Hi all, > >>> > >>> Due to huge performance issues with some of our neo queries I profiled > my > >>> calls on the neo server. The profiling shows, that up to 85% of the > time the > >>> threads are waiting for other threads. I don¹t understand what¹s going > on > >>> there. Hopefully someone with a deeper knowledge can help me. Am I > doing > >>> something wrong, or is it normal, that most of the time the threads are > >>> blocking each other? Her is a screenshot, showing the results of my > >>> profiling: http://i.imgur.com/eIfam.jpg > >>> > >>> > >>> Thanks in advice, > >>> > >>> Dario > >>> ___ > >>> Neo4j mailing list > >>> User@lists.neo4j.org > >>> https://lists.neo4j.org/mailman/listinfo/user > >>> > >> > >> > > -- > Dario Rexin > Junior Manager Engineering > dario.re...@xing.com > > XING AG > Gaensemarkt 43, 20354 Hamburg, Germany > > Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807 > Exec. Board (Vorstand): Dr. Stefan Groß-Selbeck (Vorsitzender), Ingo Chu, > Dr. Helmut Becker, Jens Pape > Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Dr. Neil > Sunderland > > Please join my network on XING: > https://www.xing.com/profile/Dario_Rexin > > This e-mail may contain confidential and/or privileged information. If you > are not the intended recipient (or have received this e-mail in error) > please notify the sender immediately and destroy this e-mail. Any > unauthorised copying, disclosure or distribution of the material in this > e-mail is strictly forbidden and may be unlawful. > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo Server slow on frequent requests
Hi Dario, Looking at that picture it is indeed clear that a number of threads are waiting for something. What is not shown is the more important information about *what* they are waiting for. I would love to get information like that in order to investigate the cause of the performance problem you are seeing. If you could send a thread dump instead of a screenshot that would be a lot more useful, since that would contain information about contention that I could actually analyze. The easiest way to get a thread dump is by sending the SIGQUIT signal (kill -3) to the JVM process running Neo4j. Cheers, Tobias On Tue, Apr 12, 2011 at 6:35 PM, Dario Rexin wrote: > Hi all, > > Due to huge performance issues with some of our neo queries I profiled my > calls on the neo server. The profiling shows, that up to 85% of the time the > threads are waiting for other threads. I don’t understand what’s going on > there. Hopefully someone with a deeper knowledge can help me. Am I doing > something wrong, or is it normal, that most of the time the threads are > blocking each other? Her is a screenshot, showing the results of my > profiling: http://i.imgur.com/eIfam.jpg > > > Thanks in advice, > > Dario > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Number of nodes/relationships visited in query?
That information is not available at the moment. But it is an interesting
idea. How would you like to access it? Can you give a quick sketch of how
you envision the API for it.
Would it be something like this?
interface PathFinder {
P findSinglePath( Node start, Node end ); // ok, nothing added here
// you'd have to use this version to get those statistics:
CountIterable findAllPaths( Node start, Node end );
}
interface CountIterable extends Iterable {
CountIterator iterator();
}
interface CountIterator extends Iterator {
int numberOfNodesVisitedSoFar();
int numberOfRelationshipsVisitedSoFar();
}
(But with more thought through names)
Since computation isn't performed until actually requested (when the
iterator is iterated over), and since the Iterable could give a different
result when you iterate over it subsequent times (due to the graph being
modified), the Iterator object is the only object where I could see that
such information could be added usefully. This does mean that you cannot use
the java foreach loop with such an Iterable AND get the visited count, but
would have to resort to using the hasNext() and next() methods. We could
quite easily add some convenience methods for making that easier though,
something like:
CountIterator pathIter = shortestPath.findAllPaths( start, end
).iterator();
for ( Path path : IterUtil.loop( pathIter ) ) {
doSomethingWith(path);
}
// after the loop is done, "number of nodes visited so far" is the total
number of visited nodes.
int visitedNodes = pathIter.numberOfNodesVisitedSoFar();
WDYT? If this is useful we could add this kind of statistics to all types of
traversals. Exposing that through the REST interface would be even simpler,
the implementation would simply do the equivalent of the iteration above
then add the statistics to the result. For paginated results (when those are
added) we could have the statistics reflect the number of nodes visited for
creating that page of data, for algorithms that find the "easiest" solutions
first, you could use those statistics to stop a search when the number of
nodes visited to collect a page grows too big.
Cheers,
Tobias
PS. I had a few other ideas on how such statistics information could be
added, but this was the least invasive of them. I sketched this up in five
minutes (after half a bottle of wine), so I'm very open to the possibility
of there being more elegant ways of adding this information.
On Sun, Apr 10, 2011 at 10:45 PM, Stephan Hagemann <
stephan.hagem...@googlemail.com> wrote:
> Hi all,
>
> is there a way for me to get to the number of nodes or relationships that
> the graph algorithms (like ShortestPath) have visited and output it with
> the
> result of the query? I would like to do that to analyze the performance and
> load of query processing. Any ideas?
>
> Thanks!
>
> Stephan
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4j Storage provider
Hi Elangovan, In theory it would be possible. Neo4j started out as an abstraction layer on top of other storage systems many years ago. The first implementation that was somewhat useable was based on Berkley DB for storage. But even then performance wasn't good enough. Not until we implemented our own storage technology was it good enough for actual use. Since then evolution of Neo4j has been progressing with our own native storage (nioneo) as the primary storage, which has made the rest of Neo4j more closely tied to that specific implementation. That said, there are still remnants of the time when the storage engine was pluggable, so it could be possible to substitute your own engine. It is going to take some patching of the internals and a fair bit of coding work, but you are welcome to try it on your own. If you don't mind me asking, why couldn't you just use Neo4j as is? That bit was a bit lost in your message. If I were you I would start out with writing a prototype of my application using Neo4j as-is. That would validate the benefits of the Neo4j graph database model, and show the benefits it would have to solving the problems that the application is meant to solve. This prototype I would then bring to my co-workers, showing them the benefits of using a graph for structuring data. First then would I start thinking of how this would fit in with the rest of the infrastructure for the application, does the benefits outweigh the cost of adding another storage engine? If the value of it is high *and* the cost of adding another storage engine is high, then I would consider adapting the model to the existing storage, but carefully being aware of the cost of doing so (i.e. losing the ability to get commercial support from the vendor, Neo Technology). I'd start with quick simple prototypes to verify to myself that the performance and other values of Neo4j didn't get lost too much to make it unfit to my needs as I adapted the model to that other storage solution. I hope that was at least the beginning of an answer to your question. I'm looking forward to your follow-up. Cheers, Tobias On Sun, Apr 10, 2011 at 3:15 AM, elangovan shanmugam wrote: > > Hi, I am new to Neo4j. Currently trying some simple graph prototypes to our > business problems. Neo4j comes up with its own storage technology. Is it > possible to use other storage technologies (say MongoDB) with Neo4j? > The main reason is that... I will be if I asked for another storage > technology in our group. I just convinced that we needed MongoDB, I would > like to see if I can leverage that with Neo4j. > Thanks & Appreciate your help. > Elangovan S > > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Trouble with Exclude and Continue in Traversal
tImpl.java:366) >at java.net.Socket.connect(Socket.java:529) >at java.net.Socket.connect(Socket.java:478) >at sun.net.NetworkClient.doConnect(NetworkClient.java:163) >at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) >at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) >at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:760) >at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632) >at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) >at > java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379) >at > com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:218) >at > com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:129) >... 14 more > > What have I done wrong? > > Thanks Mattias for helping me doing lot of the code in the Traversal > description. > > Brendan > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Lucene 3.1 with neo4j. Is it OK?
1.3 is frozen, nothing new is going in until the 1.4 line. -tobias On Thu, Apr 7, 2011 at 6:32 PM, Rick Bullotta wrote: > Any plans to officially test Lucene 3.1 for Neo 1.3 GA? > > There are plenty of useful things and bug fixes in Lucene 3.1. > > > -Original Message- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > On Behalf Of Tobias Ivarsson > Sent: Thursday, April 07, 2011 5:45 AM > To: Neo4j user discussions > Subject: Re: [Neo4j] Lucene 3.1 with neo4j. Is it OK? > > I see no reason for it not to work, Lucene has a good story of > compatibility > between releases with the same major version. > > But please note that this isn't something Neo Technology would officially > support (yet), so from that point of view you'd be "on your own". > > -t > > On Thu, Apr 7, 2011 at 11:16 AM, Balazs E. Pataki >wrote: > > > Hi, > > > > the subject says is all. :-) But anyway, is it possible/safe to use the > > brand new 3.1 (31 March 2011) release of Lucene with neo4j? > > > > Thanks, > > --- > > balazs > > _______ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > -- > Tobias Ivarsson > Hacker, Neo Technology > www.neotechnology.com > Cellphone: +46 706 534857 > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] building from source
The failures you are seeing are for tests that deal with large data. They thus need a large heap configuration to succeed. For specifying such settings for maven you would use the MAVEN_OPTS environment variable, something like: export MAVEN_OPTS=-Xmx1G Cheers, Tobias On Thu, Apr 7, 2011 at 4:58 PM, Will Holcomb wrote: > Thanks. I was trying to use the jar target rather than install. > > That build fails with: > > --- > T E S T S > --- > Running org.neo4j.com.TestCommunication > Thu Apr 07 10:53:14 EDT 2011: MadeUpServer communication server started and > bound to 1234 > [MadeUpServerProcess:27271] Thu Apr 07 10:53:15 EDT 2011: MadeUpServer > communication server started and bound to > [MadeUpServerProcess:27271] Thu Apr 07 10:53:24 EDT 2011: MadeUpServer > shutdown, closing all channels > Tests run: 5, Failures: 1, Errors: 3, Skipped: 0, Time elapsed: 10.353 sec > <<< FAILURE! > > Results : > > Failed tests: > clientGetResponseFromServerViaComLayer(org.neo4j.com.TestCommunication) > > Tests in error: > makeSureClientStoreIdsMustMatch(org.neo4j.com.TestCommunication) > makeSureServerStoreIdsMustMatch(org.neo4j.com.TestCommunication) > makeSureClientCanStreamBigData(org.neo4j.com.TestCommunication) > > Tests run: 5, Failures: 1, Errors: 3, Skipped: 0 > > > I can get it to build though with: mvn -Dmaven.test.skip=true install > > –☮W > > On Thu, Apr 7, 2011 at 1:05 AM, Peter Neubauer < > peter.neuba...@neotechnology.com> wrote: > > > Willi, > > the main build tool is maven, and we are in the middle of a > > major reshuffling on GIT, so if you install maven.apache.org and try > > "mvn clean install" that, things should work out better. Would that be > > an option? > > > > Cheers, > > > > /peter neubauer > > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Lucene 3.1 with neo4j. Is it OK?
I see no reason for it not to work, Lucene has a good story of compatibility between releases with the same major version. But please note that this isn't something Neo Technology would officially support (yet), so from that point of view you'd be "on your own". -t On Thu, Apr 7, 2011 at 11:16 AM, Balazs E. Pataki wrote: > Hi, > > the subject says is all. :-) But anyway, is it possible/safe to use the > brand new 3.1 (31 March 2011) release of Lucene with neo4j? > > Thanks, > --- > balazs > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Constructing an evaluator that only takes specific nodes from a path
That would be EXCLUDE_AND_PRUNE.
On Thu, Apr 7, 2011 at 7:58 AM, Dario Rexin wrote:
> Hi Peter, yes that would be not to include the path in the result set.
>
>
>
> On 07.04.2011, at 07:01, "Peter Neubauer" <
> peter.neuba...@neotechnology.com> wrote:
>
> > Dario,
> > I am not quite sure I understand what you mean by "drop" is that not
> > to include the path into the result or prune or something else? Do you
> > have a concrete example of this, maybe a simple toy graph test with
> > toy information?
> >
> > Cheers,
> >
> > /peter neubauer
> >
> > GTalk: neubauer.peter
> > Skype peter.neubauer
> > Phone +46 704 106975
> > LinkedIn http://www.linkedin.com/in/neubauer
> > Twitter http://twitter.com/peterneubauer
> >
> > http://www.neo4j.org - Your high performance graph
> database.
> > http://startupbootcamp.org/- Öresund - Innovation happens HERE.
> > http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.
> >
> >
> >
> > On Wed, Apr 6, 2011 at 6:35 PM, Dario Rexin
> wrote:
> >> Hello,
> >>
> >> we are trying to construct an Evaluator that will select only one node
> from the middle of a path and include it in the result. We do have a
> specific path we are looking for and the endNode is stored in 'end'. Now
> what we would like to do is have an Evaluator like this:
> >>
> >> import org.neo4j.graphdb.traversal.
> >> Evaluator;
> >>
> >> new Evaluator(){
> >>@Override
> >>public Evaluation evaluate(org.neo4j.graphdb.Path path) {
> >>if(path.length() > 3) {
> >>return Evaluation.EXCLUDE_AND_PRUNE;
> >>} else if (path.length() == 2) {
> >>return Evaluation.INCLUDE_AND_CONTINUE;
> >>} else if (path.length() < 2){
> >>return Evaluation.EXCLUDE_AND_CONTINUE;
> >>} else {
> >>return path.endNode().getId() == end.getId() ?
> Evaluation.EXCLUDE_AND_PRUNE : Evaluation.DROP_PATH_AND_PRUNE;
> >>}
> >>}
> >> }
> >>
> >> The decision on whether to accept or drop a path is based on wether we
> have reached the endNode. That's why we would like to use something like
> DROP_PATH_AND_PRUNE, which doesn't exist. What are we missing?
> >>
> >> Any help appreciated!
> >>
> >> Thanks,
> >> Stephan and Dario
> >> ___
> >> Neo4j mailing list
> >> User@lists.neo4j.org
> >> https://lists.neo4j.org/mailman/listinfo/user
> >>
> > ___
> > Neo4j mailing list
> > User@lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] CNFE in Neo4j Server
The simplest explanation is probably the correct one: it is not on the classpath. This class is to be found in the jersey-server-1.3.jar is that jar on the classpath? (or in jersey-client-1.3.jar, either will do) The simplest way one might be fooled into not including it in the classpath is due to there not being a compile-time dependency on it. the JAX-RS API does not require an implementation for compiling, but at runtime you must have a valid JAX-RS implementation available on the classpath. The default implementation is the one defined by com.sun.ws.rs.ext.RuntimeDelegateImpl, a class that jersey provides, being the reference implementation. -tobias On Thu, Apr 7, 2011 at 7:19 AM, Peter Neubauer < peter.neuba...@neotechnology.com> wrote: > Hi there, > trying to execute > > https://github.com/peterneubauer/neo4j-gremlin-plugin/blob/master/src/test/org/neo4j/server/plugin/gremlin/GremlinPluginTest.java > as a JUnit test on Eclipse/Mac with default JDK, Saikat is getting the > following: > > > java.lang.ExceptionInInitializerError > at > org.neo4j.server.rest.repr.formats.JsonFormat.(JsonFormat.java:43) > at > org.neo4j.server.plugin.gremlin.GremlinPluginTest.setUpBeforeClass(GremlinPluginTest.java:37) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27) > at org.junit.runners.ParentRunner.run(ParentRunner.java:236) > at > org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49) > at > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) > Caused by: java.lang.RuntimeException: > java.lang.ClassNotFoundException: > com.sun.ws.rs.ext.RuntimeDelegateImpl > at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:122) > at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91) > at javax.ws.rs.core.MediaType.(MediaType.java:44) > ... 17 more > Caused by: java.lang.ClassNotFoundException: > com.sun.ws.rs.ext.RuntimeDelegateImpl > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:169) > at javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:62) > at javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:155) > at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:105) > ... 19 more > > Any clues on why there is a class missing? > > Cheers, > > /peter neubauer > > GTalk: neubauer.peter > Skype peter.neubauer > Phone +46 704 106975 > LinkedIn http://www.linkedin.com/in/neubauer > Twitter http://twitter.com/peterneubauer > > http://www.neo4j.org - Your high performance graph database. > http://startupbootcamp.org/- Öresund - Innovation happens HERE. > http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] EmbeddedReadOnlyGraphDatabase workings
nce its creation. It there a way to force refresh > besides > >> > >> creating a new instance? > >> > >> > >> > >> Also are there any other/better ways to do that (1 writer, 1 > reader) > >> > >> without going into the server mode? > >> > >> > >> > >> Thanks a lot, > >> > >> > >> > >> Alfredas > >> > >> ___ > >> > >> Neo4j mailing list > >> > >> User@lists.neo4j.org > >> > >> https://lists.neo4j.org/mailman/listinfo/user > >> > > > >> > > > >> > > > >> > > -- > >> > > David Montag > >> > > Neo Technology, www.neotechnology.com > >> > > Cell: 650.556.4411 > >> > > Skype: ddmontag > >> > > > >> > > >> > >> > >> > >> -- > >> David Montag > >> Neo Technology, www.neotechnology.com > >> Cell: 650.556.4411 > >> Skype: ddmontag > >> ___ > >> Neo4j mailing list > >> User@lists.neo4j.org > >> https://lists.neo4j.org/mailman/listinfo/user > >> > > > > > > > > -- > > Mattias Persson, [matt...@neotechnology.com] > > Hacker, Neo Technology > > www.neotechnology.com > > ___ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] InvalidRecordException
This has been very thoroughly investigated, and still is being thoroughly
investigated, since we have not been able to reproduce the problem at all.
Any hints on what scenario this occurred in that you can provide would be
immensely helpful.
Cheers,
Tobias
On Tue, Apr 5, 2011 at 9:41 PM, Degrassi Francesco <
francesco.degra...@emaze.net> wrote:
> Hello everyone.
> Was this issue investigated and, perhaps, solved ?
> We started experiencing the same today, on Neo 1.1 on CentOS 5 and couldn't
> manage to reproduce it reliably yet.
>
> I will attach a stacktrace and more info tomorrow, but wanted to know if
> there
> were any news on the issue.
>
> Thanks in advance
>
> Francesco Degrassi
>
> Il 10 marzo 2011 alle 21.58 Massimo Lusetti ha
> scritto:
>
> > On Thu, Mar 10, 2011 at 6:11 PM, Axel Morgner wrote:
> >
> > > Hi,
> > >
> > > I'm getting an InvalidRecordException
> > >
> > > org.neo4j.kernel.impl.nioneo.store.InvalidRecordException: Node[5] is
> > > neither firstNode[37781] nor secondNode[37782] for Relationship[188125]
> > > at
> > >
> org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:131)
> > > at
> > >
> org.neo4j.kernel.impl.nioneo.xa.NioNeoDbPersistenceSource$ReadOnlyResourceConnection.getMoreRelationships(NioNeoDbPersistenceSource.java:280)
> > > at
> > >
> org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:100)
> > > at
> > >
> org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:585)
> > > at
> > >
> org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:358)
> > > at
> > >
> org.neo4j.kernel.impl.core.IntArrayIterator.hasNext(IntArrayIterator.java:115)
> > >
> > > when iterating through the relationships of a certain node:
> > >
> > > Node node = graphDb.getNodeById(sNode.getId());
> > >
> > > Iterable rels =
> > > node.getRelationships(relType, dir);
> > >
> > > for (Relationship r : rels) { <- here the expeption
> > > occurs
> > > ...
> > > }
> > >
> > > I'm using 1.3.M03.
> > >
> > > Seems that the database is in an inconsitant state. Don't know how this
> > > could happen ...
> > >
> > >
> > > Greetings
> > >
> > > Axel
> > >
> > > ___
> > > Neo4j mailing list
> > > User@lists.neo4j.org
> > > https://lists.neo4j.org/mailman/listinfo/user
> > >
> >
> >
> > I'm encountering this same/similar exception quite ofter lately, with
> > the same 1.3.M03 version, on FreeBSD 8.2 with OpenJDK Runtime
> > Environment (build 1.6.0-b21) OpenJDK 64-Bit Server VM (build
> > 19.0-b09, mixed mode):
> >
> > org.neo4j.kernel.impl.nioneo.store.InvalidRecordException:
> > Record[4130751] not in use
> > at
>
> >org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java:194)
> > at
>
> >org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java:96)
> > at
>
> >org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.connectRelationship(WriteTransaction.java:1435)
> > at
>
> >org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.relationshipCreate(WriteTransaction.java:1389)
> > at
>
> >org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaConnection$RelationshipEventConsumerImpl.createRelationship(NeoStoreXaConnection.java:256)
> > at
>
> >org.neo4j.kernel.impl.nioneo.xa.NioNeoDbPersistenceSource$NioNeoDbResourceConnection.relationshipCreate(NioNeoDbPersistenceSource.java:370)
> > at
>
> >org.neo4j.kernel.impl.persistence.PersistenceManager.relationshipCreate(PersistenceManager.java:153)
> > at
>
> >org.neo4j.kernel.impl.core.NodeManager.createRelationship(NodeManager.java:309)
> > at
>
> >org.neo4j.kernel.impl.core.NodeImpl.createRelationshipTo(NodeImpl.java:387)
> > at
>
> >org.neo4j.kernel.impl.core.NodeProxy.createRelationshipTo(NodeProxy.java:186)
> >
> > Did it rings and alert bell!?
> >
> > Cheers
> > --
> > Massimo
> > http://meridio.blogspot.com
> > ___
> > Neo4j mailing list
> > User@lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Query paths of certain lengths...
Hi Jeff,
What you are describing is a Friends-of-a-friend (foaf) traversal, "which
are the friends of my friends, who are not my direct friends".
The best way to perform such a traversal is by using the traversal API:
n.traverse( Order.BREADTH_FIRST, // because we want to exclude the direct
friends early
new StopEvaluator() { // don't traverse deeper than depth two
boolean isStopNode(TraversalPosition pos) { return pos.depth() == 2; }
}, new ReturnableEvaluator() { // return the ones found on depth two
boolean isReturnableNode(TraversalPosition pos) { return pos.depth()
== 2; }
}, RelTypes.FRIEND, Direction.BOTH ); // Traverse Friend relationships,
in any direction
Since this traverses breadth first, it will find the direct friends first,
and exclude those, since this kind of traversal does not revisit nodes.
The traversal above almost does what you want, except that it only returns
nodes, and you wanted the paths. For that you can use our more capable
tentative traversal API to do the same traversal, but getting paths instead:
TraversalDescription foaf = Traversal.description()
.breadthFirst()
.relationships( RelTypes.FRIEND, Direction.OUTGOING )
.evaluator( new Evaluator() {
Evaluation evaluate( Path path )
{
return ( path.length() == 2 ) ? Evaluation.INCLUDE_AND_PRUNE :
Evaluation.EXCLUDE_AND_CONTINUE;
}
} );
Then you'd just iterate over the paths from the given node:
for (Path foafPath : foaf.traverse( n ) ) {
printPath( foafPath );
}
If you want that for ALL possible start-nodes, I guess you would have to
iterate through all nodes and apply the same traversal. But given just a
single start node, the above traversal is enough, and it is fast enough to
do in real time (show the friends of my friends on a social media website).
Cheers,
Tobias
On Wed, Mar 30, 2011 at 7:51 PM, jisenhart wrote:
>
> Suppose I have the following node/paths
>
> n -> n1
> n -> n2
> n -> n4
>
> n2 -> n1
> n2 -> n3
> n2 -> n4
>
> I want to find all paths of depth two (for example):
>
> n -> n2 -> n3
> n -> n2 -> n4
>
> and filter out those paths where a shorter path exists to a given node
> (n) leaving just
>
> n -> n2 -> n3
>
> since n -> n4 is "shallower" than n -> n2 -> n4
>
>
> Is this possible? I see GraphAlgoFactory.pathsWithLength(expander,
> length). But unclear on how to proceed beyond that.
>
> Jeff
>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Glassfish problem
Hi Matěj, As the error message says, most likely this is caused by another instance already being open in the same JVM. If I were to have I guess, I think that your code (in the singleton you mention) create a new GraphDatabaseService without invoking shutdown() on the existing one first. Probably what you've done is just null out the reference (which is why your code thinks it has to start a new instance). There is no need to startup and shutdown the graph database multiple times (are you doing this for each request?), it is not a database connection, it is the database, and startup and shutdown takes time. It is better to tie the lifecycle of the graph database to the lifecycle of your application. Cheers, Tobias 2011/3/28 Matěj Plch > Hi! So we deployed our application, but we get Caused by: > java.lang.IllegalStateException: Unable to lock store ... SEVERE: > 0-database/neostore], this is usually caused by another Neo4j kernel > already running in this JVM for this particular store > Is there any best practise how to avoid this? The database is opened or > created from class with is Singleton. Can this be the problem? > I googled this problem, but no solution found. > Matěj Plch > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] A picture worth a 170'000 nodes and 650'000 edges
Nice to see my visualization code being put to good use. The most recent version has moved to github, along with the rest of Neo4j, since this is an unofficial tool it is on my personal github: https://github.com/thobe/neoviz Cheers, Tobias On Thu, Mar 24, 2011 at 8:09 PM, Alfredas Chmieliauskas wrote: > For fun: > > This is a visualization of the graph resulting from a simulation > (interacting energy markets). > > http://test.eeni.tbm.tudelft.nl/~alfredas/d13n-graph.png > > After 10 simulation ticks we have ~170'000 entities (agents, markets, > power plants, bids, substances, technologies, etc) and ~650'000 > relations between these entities. Different colors represent different > types of relations. These relations are created at a rate of ~60'000 > per tick, as a result of agents trading, investing and energy flows. > By the end of the simulation we get ~1m nodes and ~4m edges. > > The visualization was made using neo-graphviz: > https://svn.neo4j.org/laboratory/components/neo-graphviz/ > > The crazy part is that we can still make sense of what's happening > there, using a few pipes and traversals :-) > > Alfredas > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Filter by relationship Property
For the first position (at the start node), there is no relationship leading
up to it, which means that the lastRelationship() method returns null.
This is a special case that has to be dealt with.
Cheers,
Tobias
On Wed, Mar 23, 2011 at 9:05 PM, Justine Mathews wrote:
> Tobias,
>
>
>
> I have used the return filter, I am getting following error when making
> rest call for node traverse,
>
> Error : The remote server returned an error: (500) Internal Server Error.
>
> Response header status : *javax.script.ScriptException:
> sun.org.mozilla.javascript.internal.EcmaError: TypeError: Cannot call method
> "getProperty" of null (#1)*
>
>
>
> please find below the syntax,
>
>
>
> *url:* http://localhost:7474/db/data/node/80/traverse/node
>
> *traverse:*
>
> {"order": "depth first","relationships": [
>
> {
>
> "type": "Friend",
>
> "direction": "all"
>
> }
>
> ],"return filter": {
>
> "language": "javascript",
>
> "body": "position.lastRelationship().hasProperty(\"customerId\");"
>
> },"max depth": 1}
>
>
>
> · *Kernel version*
>
> Neo4j - Graph Database Kernel 1.3.M03
>
>
>
>
>
> --
> Regards,
> Justine K Mathews, MCSD.NET
> Mob: +44-(0) 7795268546
>
> http://www.justinemathews.com
>
> http://uk.linkedin.com/in/justinemathews
>
>
>
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] 回复： neo4j reade with getNodeById && index.with...
Why not just keep the reference to the actual Node object then? Neo4j's
internal cache and memory management makes this really cheap to do. [1]
As to your questions about size of property of an index and maximum memory,
you will have to be clearer, I really don't understand what you are asking
for.
-tobias
[1] a Node object always occupies exactly 32B in a 64bit JVM, and I believe
that would translate to 20B on a 32bit JVM. The actual state of the Node is
stored internally in Neo4j and managed through the Neo4j cache.
2011/3/23 孤竹
> Yes, I know I can't assign the IDS. But I can got some node's ID ,and cache
> it in some place( i.e. in the cache).
>
>
> when I need to search something , I got the id in the cache . Is it better
> ?
>
>
> At last, I wonder what size is property of the index? Depends On the memory
> I gived to Neo4j ?
>
>
> Is there a suggest rate ? for example , If the max-memory for Neo4j is 1G,
> and the index is better less than 500M ?
>
>
> -- 原始邮件 --
> 发件人: "mattias";
> 发送时间: 2011年3月23日(星期三) 下午4:24
> 收件人: "Neo4j user discussions";
>
> 主题: Re: [Neo4j] neo4j reade with getNodeById && index.with...
>
>
> And also to circumvent this different in performance for not-very-big
> indexes you can make use of the built-in caching (for the new API)... see
>
> http://docs.neo4j.org/chunked/snapshot/indexing-lucene-extras.html#indexing-lucene-caching
>
> 2011/3/23 Tobias Ivarsson
>
> > It is correct that getNodeById is much faster than an index lookup, but
> IDs
> > are assigned by Neo4j, there is no way for you as a user to assign IDs,
> > which makes it a very blunt tool for looking up entities. To know which
> id
> > corresponds to a particular name or similar attribute, you would have to
> go
> > through an index, and that is exactly what the Neo4j index API does for
> > you.
> >
> > By the way, the LuceneIndexService is a deprecated API, use
> > GraphDatabaseService#index() instead:
> >
> >
> http://components.neo4j.org/neo4j/1.3.M04/apidocs/org/neo4j/graphdb/GraphDatabaseService.html#index()
> >
> > Cheers,
> > Tobias
> >
> > On Wed, Mar 23, 2011 at 3:45 AM, 孤竹 wrote:
> >
> > > HI,all
> > >
> > > I Found something interesting, The test as follow:
> > >
> > > public static void main(String args[]) {
> > >GraphDatabaseService db = new EmbeddedGraphDatabase("testDB");
> > >indexService = new LuceneIndexService(db);
> > >Long startTime = System.currentTimeMillis();
> > >for(int i =0 ;i <1;i++){
> > >Node node = getNodeByIndexProperty(db,String.valueOf(i));
> > >}
> > >Long endTime = System.currentTimeMillis();
> > >System.out.println("time = " + (endTime - startTime));
> > >}
> > >public static Node getNodeByIndexProperty(GraphDatabaseService db,
> > > String keyValue) {
> > > //Node node = indexService.getSingleNode(KEY_NAME, keyValue);
> > >Node node = db.getNodeById(Long.valueOf(keyValue));
> > >return node;
> > >}
> > >
> > > When I try to use the method indexService.getSingleNode.. with
> > > times, It will take 2 seconds (nearly) , BY when I use
> db.getNodeById
> > > same times, It just take 350 millisecond . Does it mean getById is
> faster
> > > than use index ? or Does that mean the natural index is better than
> index
> > if
> > > I can use it ?
> > >
> > > thanks for your help ,and replies, that help me very much!
> > > ___
> > > Neo4j mailing list
> > > User@lists.neo4j.org
> > > https://lists.neo4j.org/mailman/listinfo/user
> > >
> >
> >
> >
> > --
> > Tobias Ivarsson
> > Hacker, Neo Technology
> > www.neotechnology.com
> > Cellphone: +46 706 534857
> > ___
> > Neo4j mailing list
> > User@lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
> >
>
>
>
> --
> Mattias Persson, [matt...@neotechnology.com]
> Hacker, Neo Technology
> www.neotechnology.com
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j reade with getNodeById && index.with...
It is correct that getNodeById is much faster than an index lookup, but IDs
are assigned by Neo4j, there is no way for you as a user to assign IDs,
which makes it a very blunt tool for looking up entities. To know which id
corresponds to a particular name or similar attribute, you would have to go
through an index, and that is exactly what the Neo4j index API does for you.
By the way, the LuceneIndexService is a deprecated API, use
GraphDatabaseService#index() instead:
http://components.neo4j.org/neo4j/1.3.M04/apidocs/org/neo4j/graphdb/GraphDatabaseService.html#index()
Cheers,
Tobias
On Wed, Mar 23, 2011 at 3:45 AM, 孤竹 wrote:
> HI,all
>
> I Found something interesting, The test as follow:
>
> public static void main(String args[]) {
>GraphDatabaseService db = new EmbeddedGraphDatabase("testDB");
>indexService = new LuceneIndexService(db);
>Long startTime = System.currentTimeMillis();
>for(int i =0 ;i <1;i++){
>Node node = getNodeByIndexProperty(db,String.valueOf(i));
>}
>Long endTime = System.currentTimeMillis();
>System.out.println("time = " + (endTime - startTime));
>}
>public static Node getNodeByIndexProperty(GraphDatabaseService db,
> String keyValue) {
> //Node node = indexService.getSingleNode(KEY_NAME, keyValue);
>Node node = db.getNodeById(Long.valueOf(keyValue));
>return node;
>}
>
> When I try to use the method indexService.getSingleNode.. with
> times, It will take 2 seconds (nearly) , BY when I use db.getNodeById
> same times, It just take 350 millisecond . Does it mean getById is faster
> than use index ? or Does that mean the natural index is better than index if
> I can use it ?
>
> thanks for your help ,and replies, that help me very much!
> _______
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Filter by relationship Property
Hi Justine,
What you have specified is to prune traversal branches where customerId is
2, this means that the traversal will not continue further from that point,
but it doesn't mean that this position will not be returned. For that you
need to specify a "return filter" instead of or in addition to the "prune
evaluator".
The syntax for "return filter" is the same as for "prune evaluator", see:
http://components.neo4j.org/neo4j-server/stable/rest.html#Traverse
Cheers,
Tobias
On Tue, Mar 22, 2011 at 11:55 PM, Justine Mathews <
justine.math...@megree.com> wrote:
>
> Please find below the traversal syntax I am using from rest client, The
> result is not filtered by the
> customerId( relationship property) value 2.
>
> {"order": "depth first","relationships": [
> {
>"type": "Friend",
>"direction": "all"
> }
> ],"prune evaluator": {
> "language": "javascript",
> "body": "position.lastRelationship().getProperty(\"customerId\") == 2;"
> }}
>
>
>
> --
> Regards,
> Justine K Mathews, MCSD.NET
> Mob: +44-(0) 7795268546
> http://www.justinemathews.com<http://www.justinemathews.com/>
> http://uk.linkedin.com/in/justinemathews
>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Fans of Neo4j From Chinese
Rick, I don't quite understand what you are asking for. Could you elaborate
further?
-tobias
2011/3/22 Rick Bullotta
> I'd like to explore this question a bit further. Does this mean that
> basically there's no way to scale beyond a single thread/CPU for
> disconnected graphs if you have complex graph dependencies (e.g. you cannot
> create disjoint subgraphs)?
>
>
>
> -Original Message-
> From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org]
> On Behalf Of Tobias Ivarsson
> Sent: Saturday, March 19, 2011 5:59 AM
> To: Neo4j user discussions
> Subject: Re: [Neo4j] Fans of Neo4j From Chinese
>
> Neo4j serializes commits. I.e. at most one thread is committing a
> transaction at once.
> For the actual work of building up the data to be committed, Neo4j supports
> multiple concurrent threads.
>
> This fact alone, that there is a single congestion point, means that if an
> application, like in your case, is very write centric, it is unlikely for
> it
> to scale beyond two threads, with one building up the next commit while the
> other is commiting its data. It might scale to a few more threads than that
> if the buildup time is significantly larger than the commit time. It is
> simple time slicing, "only one train can be at the station at once", then
> you have to do the maths on how many "trains can be out on the track"
> during
> that time.
>
> It is also worth keeping in mind, that for CPU bound operation, an
> application doesn't scale much further than the number of CPUs in the
> computer. The threads that are not in commit mode - i.e. the ones that are
> building up the data for their next commit - are CPU bound, and contending
> for the same CPU resources. This means that your application is not going
> to
> scale much further than the number of CPUs in your computer, and few
> desktop/laptop computers have more than 4 CPUs these days, which makes 5
> threads about the most you can squeeze out of it, anything more than that
> is
> just going to add contention, and possibly even slow things down.
>
> Finally, the (CPU bound) threads that create the graph might be contending
> on the same resources. As Peter said. If multiple threads modify the same
> node or relationship, i.e. if they create relationships to the same node
> (the root node for example), they are all going to block on that resource.
> Neo4j only allows one transaction to modify each entity at a time. This
> means that to get maximum concurrency out of your data creation, each
> thread
> should be creating each own disconnected subgraph. And if they have
> connected parts, the connections to the "global data" should be made last
> in
> the transaction (in a predictable order to avoid deadlocks[1]), to maximize
> the time the thread is operational before hitting the
> "congestion point" that is the (potentially) contended data.
>
> Cheers,
> Tobias
>
> [1] Neo4j will detect if a deadlock has occurred and throw a
> DeadlockDetectedException in that case.
>
> 2011/3/18 孤竹
>
> > hi,
> >
> >
> > Sorry for disturb you , I am a chinese engineer , Excused for my bad
> > english :) .
> >
> >
> > Recently, I am learning Neo4j and trying to use it in my project . But
> > When I make a Pressure on neo4j with 5 theads , 10 theads, 20 and 30, I
> > found the nodes inserted to the Neo4J is not change obvious (sometimes
> not
> > change ~ ~! ). Does it not matter with threads ? the kenerl will make it
> > Serial ? Is there any documents or something about The performance of
> Neo4j
> > ? thanks for your help
> >
> >
> >
> > The program as follows:
> > I put this function in ExecutorService ,with 5/10/30 threads. then test
> > for the nodes inserted into at same time .(The counts have not changed
> > obviously)
> >
> >
> > Transaction tx = null;
> >Node before = null;
> >try {
> >for (int i = 0; i < 100; i++) {
> >if(stop == true){
> >return;
> >}
> >if (graphDb == null) {
> >return;
> >}
> >try {
> >if (tx == null) {
> >tx = graphDb.beginTx();
> >}
> >
Re: [Neo4j] Fans of Neo4j From Chinese
gt; KNOWS);
>before = end;
>// 每一千次 commit一次
>if (i % 100 == 0) {
>tx.success();
>tx.finish();
>tx = null;
>}
>} catch (Exception e) {
> System.out.println("write : = " +
> e);
>}
>}
>} catch (Exception e) {
>} finally {
>tx.finish();
>}
>}
>
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Problems installing neo4j.py on Mac
neo4j-python/src/site/site.xml > Aneo4j-python/src/site/apt > Aneo4j-python/src/site/apt/apidoc > Aneo4j-python/src/site/apt/apidoc/index.apt > Aneo4j-python/src/examples > Aneo4j-python/src/examples/python > Aneo4j-python/src/examples/python/djangosites > Aneo4j-python/src/examples/python/djangosites/manage.py > Aneo4j-python/src/examples/python/djangosites/faces > Aneo4j-python/src/examples/python/djangosites/faces/views.py > Aneo4j-python/src/examples/python/djangosites/faces/__init__.py > Aneo4j-python/src/examples/python/djangosites/faces/tests.py > Aneo4j-python/src/examples/python/djangosites/faces/models.py > Aneo4j-python/src/examples/python/djangosites/faces/urls.py > Aneo4j-python/src/examples/python/djangosites/blog > Aneo4j-python/src/examples/python/djangosites/blog/views.py > Aneo4j-python/src/examples/python/djangosites/blog/__init__.py > Aneo4j-python/src/examples/python/djangosites/blog/tests.py > Aneo4j-python/src/examples/python/djangosites/blog/models.py > Aneo4j-python/src/examples/python/djangosites/blog/urls.py > Aneo4j-python/src/examples/python/djangosites/__init__.py > Aneo4j-python/src/examples/python/djangosites/settings.py > Aneo4j-python/src/examples/python/djangosites/imdb > Aneo4j-python/src/examples/python/djangosites/imdb/orm_model.py > Aneo4j-python/src/examples/python/djangosites/imdb/views.py > Aneo4j-python/src/examples/python/djangosites/imdb/__init__.py > Aneo4j-python/src/examples/python/djangosites/imdb/importer.py > Aneo4j-python/src/examples/python/djangosites/imdb/neo_model.py > Aneo4j-python/src/examples/python/djangosites/imdb/data > A > neo4j-python/src/examples/python/djangosites/imdb/data/test-actors.list > Aneo4j-python/src/examples/python/djangosites/imdb/data/sample.py > Aneo4j-python/src/examples/python/djangosites/imdb/data/actors.txt > A > neo4j-python/src/examples/python/djangosites/imdb/data/test-movies.list > Aneo4j-python/src/examples/python/djangosites/imdb/tests.py > Aneo4j-python/src/examples/python/djangosites/imdb/models.py > Aneo4j-python/src/examples/python/djangosites/imdb/templatetags > A > neo4j-python/src/examples/python/djangosites/imdb/templatetags/__init__.py > A > neo4j-python/src/examples/python/djangosites/imdb/templatetags/apply.py > Aneo4j-python/src/examples/python/djangosites/imdb/urls.py > Aneo4j-python/src/examples/python/djangosites/urls.py > Aneo4j-python/src/examples/python/djangosites/templates > A > neo4j-python/src/examples/python/djangosites/templates/list_of_links.html > Aneo4j-python/src/examples/python/djangosites/templates/imdb > A > neo4j-python/src/examples/python/djangosites/templates/imdb/parts_table.html > A > neo4j-python/src/examples/python/djangosites/templates/imdb/list_view.html > Aneo4j-python/src/examples/python/djangosites/templates/imdb/index.html > A > neo4j-python/src/examples/python/djangosites/templates/imdb/searchform.html > Aneo4j-python/src/examples/python/djangosites/templates/imdb/view.html > Aneo4j-python/src/examples/python/djangosites/templates/imdb/list.html > Aneo4j-python/src/examples/python/djangosites/templates/base.html > Aneo4j-python/pom.xml > Aneo4j-python/README.txt > Aneo4j-python/tech-debt.txt > Exported revision 8590. > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4j OSGI bundle
Oh, right. Only the bundles that org.neo4j:neo4j depends on exports packages. I guess org.neo4j:neo4j would have to re-export the packages it depends on, so that your application can depend on org.neo4j:neo4j and use those packages. Is that the appropriate way? I have little OSGi experience (other than cursing over it when it fails the build process because you need to very explicitly declare which packages to include). If that would solve things for you we will make sure to make those changes for the next milestone. In the meantime you can directly depend on the bundles org.neo4j:neo4j depends on, you are probably interested in at least: org.neo4j:neo4j-kernel org.neo4j:neo4j-lucene-index But perhaps these as well: org.neo4j:neo4j-management - JMX support org.neo4j:neo4j-graph-algo - Graph algorithms, such as shortest path org.neo4j:neo4j-com - Communication support, this includes the online-backup server, and the bases that HA depends on org.neo4j:neo4j-ha - High availability support for Neo4j You might also want to be nice to us and have Neo4j send a ping to us every day, letting us know how many people use Neo4j: org.neo4j:neo4j-udc Please let us know how this works out for you, and if there are any problems. We have only done casual testing with OSGi, so any feedback is much appreciated. Cheers, Tobias On Fri, Mar 11, 2011 at 10:40 PM, Nicolas Jouanin wrote: > Hi, > > I'm planning to use Neo4j under OSGi framework. Because org.neo4j:neo4j > defines the needed headers (in MANIFEST.MF), it can be deployed as a bundle > on equinox or felix. > Unfortunatelly, the bundle doesn't export any package so it's quite > unuseful for me as my own bundle can't import them. > I guess, at least neo4j bundle should have Export-Package directive in > MANIFEST.MF file. > Do you think these headers could be added in the next snapshot ? > > Thanks. > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Announcing Neo4j 1.3 “Abisko Lampa” M04
It is in git: https://github.com/neo4j/graphdb Neo4j moved from svn to git a little over a month ago. Cheers, Tobias On Fri, Mar 11, 2011 at 3:21 PM, Jonah H. Harris < jonah.har...@myyearbook.com> wrote: > On Fri, Mar 11, 2011 at 9:02 AM, Chris Gioran < > chris.gio...@neotechnology.com> wrote: > > > The most important feature of this release is of course the expansion of > > the > > storage layer to allow for even more entities, > > > Hey Chris, > > This sounds great. Is it checked into SVN? I'd like to see the > differences > in the file formats. > > -- > Jonah H. Harris, VP of Database Administration > myYearbook.com > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] InvalidRecordException
I need at least these two files: neostore.nodestore.db neostore.relationshipstore.db -tobias On Fri, Mar 11, 2011 at 11:26 AM, Massimo Lusetti wrote: > On Fri, Mar 11, 2011 at 10:49 AM, Tobias Ivarsson > wrote: > > > Please tell me that you have the database file in the state they were in > > when this happened. That you have not tried to repair the database in any > > way. > > If you do, could you please send me those database files off-list > (dropbox > > or similar, or contact me directly to discuss how to transfer it). > > Sure I have it unmodified and could make it available to you with > dropbox but the db dir it's 2G... so is there something I could leave > out? > If you prefer you can continue to mail privately... > > > Massimo: If you can produce a test case that reliably (or at least > > semi-reliably) reproduces this, that would of course be even better, but > it > > doesn't hurt if I could look at the store files and try to deduce the > > problem in parallell. > > I'm working on it... will let you know soon. > > Thanks for the interest... Cheers > -- > Massimo > http://meridio.blogspot.com > _______ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] InvalidRecordException
Please tell me that you have the database file in the state they were in when this happened. That you have not tried to repair the database in any way. If you do, could you please send me those database files off-list (dropbox or similar, or contact me directly to discuss how to transfer it). I will of course treat your data with utmost privacy, but I would love to get to examine a store that exhibits this behavior to find out what could have caused it. Massimo: If you can produce a test case that reliably (or at least semi-reliably) reproduces this, that would of course be even better, but it doesn't hurt if I could look at the store files and try to deduce the problem in parallell. Cheers, -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] doInternalRecovery takes a long time
J2SE 5.0 reached its End of Service Life (EOSL) on November 3, 2009, which is the date of the final publicly available update of version 5.0 (J2SE 5.0 Update 22). See: http://www.oracle.com/technetwork/java/javase/downloads/index-jdk5-jsp-142662.html We have decided to not support a JDK that is no longer supported by the JDK vendor. Compiling our code with the class file version set to 1.6 is our way to *strongly* urge you to move to JDK 1.6, which in every way is a better JDK. If you for some reason still need to use JDK 1.5, you could build your own, unsupported, Neo4j version. It should be possible to compile Neo4j using JDK 1.6 but setting the target version to 1.5. Most of the strict 1.6 dependencies are compile-time only. There might be a few runtime dependencies on 1.5, but those are probably limited to java.util.Deque and java.util.concurrent.TimeUnit. Your other option is to get a payed JDK 1.5 support contract for Neo4j from Neo Technology. Cheers, Tobias On Thu, Mar 10, 2011 at 8:54 AM, Samuel Feng wrote: > David, > > I am using jdk1.5. > > After upgraded to 1.3M03, I Encountered* * > *java.lang.UnsupportedClassVersionError: Bad version number in .class file* > Error. > > Seems the jars in neo4j's lib folder are complied using jdk6. > > Can I use 1.5 to compiled the sources directly? > > > 2011/3/10 Samuel Feng > > > Thanks David. > > > > My concern is if the neo4j data in 1.2 can be used in 1.3 directly? > > > > I have used custom analyzer in my index. > > > > 2011/3/10 David Montag > > > > Samuel, > >> > >> Can you try it with 1.3.M03, if possible? > >> > >> Thanks, > >> David > >> > >> On Wed, Mar 9, 2011 at 6:13 PM, Samuel Feng wrote: > >> > >>> Dear list, > >>> > >>> I am developing a tomcat application in eclipse which has about 100,000 > >>> nodes using EmbeddedGraphDatabase (Neo4j version 1.2M05) > >>> > >>> Sometimes I took me a long time(more than half an hour) to > >>> doInternalRecovery, maybe because I shutdown the tomcat server > directly. > >>> > >>> From the messages.log, I can find many many "Injected two phase > commit", > >>> *Can > >>> u tell me what cause this and how to prevent it?* > >>> > >>> Thu Mar 10 09:47:47 CST 2011: Opened > >>> [C:\home\heartwater\graph\\nioneo_logical.log.1] clean empty log, > >>> version=1 > >>> Thu Mar 10 09:47:47 CST 2011: Opened > >>> [C:\home\heartwater\graph\/lucene/lucene.log.1] clean empty log, > >>> version=0 > >>> Thu Mar 10 09:47:47 CST 2011: Opened > >>> [C:\home\heartwater\graph\/lucene-fulltext/lucene.log.1] clean empty > log, > >>> version=0 > >>> Thu Mar 10 09:47:47 CST 2011: Non clean shutdown detected on log > >>> [C:\home\heartwater\graph\index/lucene.log.1]. Recovery started ... > >>> Thu Mar 10 09:47:47 CST 2011: > >>> [C:\home\heartwater\graph\index/lucene.log.1] > >>> logVersion=0 with committed tx=1 > >>> Thu Mar 10 09:47:48 CST 2011: Injected two phase commit, txId=2 > >>> Thu Mar 10 09:47:48 CST 2011: Injected two phase commit, txId=3 > >>> Thu Mar 10 09:47:48 CST 2011: Injected two phase commit, txId=4 > >>> Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=5 > >>> Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=6 > >>> Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=7 > >>> Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=8 > >>> Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=9 > >>> Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=10 > >>> Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=11 > >>> Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=12 > >>> Thu Mar 10 09:47:50 CST 2011: Injected two phase commit, txId=13 > >>> Thu Mar 10 09:47:50 CST 2011: Injected two phase commit, txId=14 > >>> ___ > >>> Neo4j mailing list > >>> User@lists.neo4j.org > >>> https://lists.neo4j.org/mailman/listinfo/user > >>> > >> > >> > >> > >> -- > >> David Montag > >> Neo Technology, www.neotechnology.com > >> Cell: 650.556.4411 > >> david.mon...@neotechnology.com > >> > > > > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How to copy a complete database?
That class is in Neo4j kernel. It is the actual implementation class. You can find the source code here: https://github.com/neo4j/graphdb/blob/master/kernel/src/main/java/org/neo4j/kernel/impl/nioneo/store/ShortString.java All of the classes in the org.neo4j.kernel.impl.nioneo.store package of my neo4j-admin-store project are of that kind, facade classes that provide access to internals of the neo4j kernel implementation. Cheers, Tobias On Mon, Mar 7, 2011 at 1:00 PM, Balazs E. Pataki wrote: > Hi Tobias, > > I tried to use ShortStringEncoding.java, however it seems to refer to a > class called ShortString, which is not available in github: > > > https://github.com/thobe/neo4j-admin-store/tree/master/src/main/java/org/neo4j/kernel/impl/nioneo/store > > Could you also please provide it? > > Thanks, > --- > balazs > > On 3/4/11 5:05 PM, Tobias Ivarsson wrote: > > Hi Balazs, > > > > I have created such a thing, the tool is available here: > > > https://github.com/downloads/thobe/neo4j-admin-store/propstat-1.3.M03.jar > > Run it as: java -jar propstat-1.3.M03.jar > > > > This tool will gather statistics about all of your properties, and output > > how many you have of each kind. > > For short string properties it will output how many you have in each > > encoding. > > For regular string properties it will output how many could have been > stored > > as short strings, and with which encoding that would be. > > > > That jar file contains a class that can be used for testing if a given > > string is "short" or not, and how it would be stored if it was considered > > short: > > > https://github.com/thobe/neo4j-admin-store/blob/master/src/main/java/org/neo4j/kernel/impl/nioneo/store/ShortStringEncoding.java > > The method you are interested in is (the first one) getEncoding(String), > it > > returns one of the Enum constants in that class, or null if the provided > > string would not be considered "short". > > > > While I wrote this I realized that it would be really simple to write a > tool > > that converts all strings that are short enough to be stored as short > > strings, that tool is available here: > > > https://github.com/downloads/thobe/neo4j-admin-store/makeshort-1.3.M03.jar > > Run it as: java -jar makeshort-1.3.M03.jar > > > > This tool will modify your string properties to store them as short > strings > > when possible. It will then rebuild the id generator for the > > DynamicStringStore so that the "holes" created where the strings were > > previously stored will be reused the next time you write string > properties > > to Neo4j (unless they are also short of course). > > > > Please note that both of these tools come with no official support or > > guarantees. Use at your own risk. > > > > Cheers, > > Tobias > > > > On Fri, Mar 4, 2011 at 10:19 AM, Balazs E. Pataki >wrote: > > > >> Actually both options would be very useful, but the first option (string > >> as input and tell me if it is short or not) would be enough for me for > >> now if available. > >> > >> Thanks, > >> --- > >> balazs > >> > >> On 3/3/11 6:37 PM, Tobias Ivarsson wrote: > >>> Yes, the statistics tool almost does that, but with a wider range of > >>> possible encodings. > >>> > >>> What would you want from a class like that? Give it a string as input > and > >>> have it tell you if it is short or not? Or go through an existing store > >> and > >>> tell you how many of the strings are short? > >>> > >>> -tobias > >>> > >>> On Thu, Mar 3, 2011 at 5:18 PM, Balazs E. Pataki >>> wrote: > >>> > >>>> Hi Tobias, > >>>> > >>>> Do you happen to have some little class which can at tell us which > >>>> property values neo4j would consider short and which of your 6 > encoding > >>>> would it use? Maybe the string statistics collecting program you > created > >>>> earlier has this logic - but I'm just guessing. > >>>> > >>>> Thanks, > >>>> --- > >>>> balazs > >>>> > >>>> On 3/3/11 11:52 AM, Tobias Ivarsson wrote: > >>>>> No there is no simpler way, yet. We've been thinking about creating a > >>>> "short > >>>>> string compression tool" for accomplishing this, but haven't done so > >>
Re: [Neo4j] How to copy a complete database?
Hi Balazs, I have created such a thing, the tool is available here: https://github.com/downloads/thobe/neo4j-admin-store/propstat-1.3.M03.jar Run it as: java -jar propstat-1.3.M03.jar This tool will gather statistics about all of your properties, and output how many you have of each kind. For short string properties it will output how many you have in each encoding. For regular string properties it will output how many could have been stored as short strings, and with which encoding that would be. That jar file contains a class that can be used for testing if a given string is "short" or not, and how it would be stored if it was considered short: https://github.com/thobe/neo4j-admin-store/blob/master/src/main/java/org/neo4j/kernel/impl/nioneo/store/ShortStringEncoding.java The method you are interested in is (the first one) getEncoding(String), it returns one of the Enum constants in that class, or null if the provided string would not be considered "short". While I wrote this I realized that it would be really simple to write a tool that converts all strings that are short enough to be stored as short strings, that tool is available here: https://github.com/downloads/thobe/neo4j-admin-store/makeshort-1.3.M03.jar Run it as: java -jar makeshort-1.3.M03.jar This tool will modify your string properties to store them as short strings when possible. It will then rebuild the id generator for the DynamicStringStore so that the "holes" created where the strings were previously stored will be reused the next time you write string properties to Neo4j (unless they are also short of course). Please note that both of these tools come with no official support or guarantees. Use at your own risk. Cheers, Tobias On Fri, Mar 4, 2011 at 10:19 AM, Balazs E. Pataki wrote: > Actually both options would be very useful, but the first option (string > as input and tell me if it is short or not) would be enough for me for > now if available. > > Thanks, > --- > balazs > > On 3/3/11 6:37 PM, Tobias Ivarsson wrote: > > Yes, the statistics tool almost does that, but with a wider range of > > possible encodings. > > > > What would you want from a class like that? Give it a string as input and > > have it tell you if it is short or not? Or go through an existing store > and > > tell you how many of the strings are short? > > > > -tobias > > > > On Thu, Mar 3, 2011 at 5:18 PM, Balazs E. Pataki >wrote: > > > >> Hi Tobias, > >> > >> Do you happen to have some little class which can at tell us which > >> property values neo4j would consider short and which of your 6 encoding > >> would it use? Maybe the string statistics collecting program you created > >> earlier has this logic - but I'm just guessing. > >> > >> Thanks, > >> --- > >> balazs > >> > >> On 3/3/11 11:52 AM, Tobias Ivarsson wrote: > >>> No there is no simpler way, yet. We've been thinking about creating a > >> "short > >>> string compression tool" for accomplishing this, but haven't done so > yet. > >>> > >>> Cheers, > >>> Tobias > >>> > >>> On Thu, Mar 3, 2011 at 11:35 AM, Balazs E. Pataki >>> wrote: > >>> > >>>> Hi, > >>>> > >>>> I have a big database based on Neo4J 1.2. Now, if I would like to use > >>>> the "short strings" feature of Neo4j 1.3 M03 I should regenerate my > full > >>>> database, that is all strings should be reset so that it may or may > not > >>>> be stored according to the new "short strings" policy. > >>>> > >>>> It seems to me that the easiest way to do this would be to somehow be > >>>> able to copy the full 1.2 database to a newly created 1.3 M03 database > >>>> by traversing the 1.2 database. But there maybe a simpler (neo4j > >>>> builtin) way to do this. Any hints about this? > >>>> > >>>> Thanks, > >>>> --- > >>>> balazs > >>>> ___ > >>>> Neo4j mailing list > >>>> User@lists.neo4j.org > >>>> https://lists.neo4j.org/mailman/listinfo/user > >>>> > >>> > >>> > >>> > >> ___ > >> Neo4j mailing list > >> User@lists.neo4j.org > >> https://lists.neo4j.org/mailman/listinfo/user > >> > > > > > > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How to copy a complete database?
Yes, the statistics tool almost does that, but with a wider range of possible encodings. What would you want from a class like that? Give it a string as input and have it tell you if it is short or not? Or go through an existing store and tell you how many of the strings are short? -tobias On Thu, Mar 3, 2011 at 5:18 PM, Balazs E. Pataki wrote: > Hi Tobias, > > Do you happen to have some little class which can at tell us which > property values neo4j would consider short and which of your 6 encoding > would it use? Maybe the string statistics collecting program you created > earlier has this logic - but I'm just guessing. > > Thanks, > --- > balazs > > On 3/3/11 11:52 AM, Tobias Ivarsson wrote: > > No there is no simpler way, yet. We've been thinking about creating a > "short > > string compression tool" for accomplishing this, but haven't done so yet. > > > > Cheers, > > Tobias > > > > On Thu, Mar 3, 2011 at 11:35 AM, Balazs E. Pataki >wrote: > > > >> Hi, > >> > >> I have a big database based on Neo4J 1.2. Now, if I would like to use > >> the "short strings" feature of Neo4j 1.3 M03 I should regenerate my full > >> database, that is all strings should be reset so that it may or may not > >> be stored according to the new "short strings" policy. > >> > >> It seems to me that the easiest way to do this would be to somehow be > >> able to copy the full 1.2 database to a newly created 1.3 M03 database > >> by traversing the 1.2 database. But there maybe a simpler (neo4j > >> builtin) way to do this. Any hints about this? > >> > >> Thanks, > >> --- > >> balazs > >> _______ > >> Neo4j mailing list > >> User@lists.neo4j.org > >> https://lists.neo4j.org/mailman/listinfo/user > >> > > > > > > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How to copy a complete database?
No there is no simpler way, yet. We've been thinking about creating a "short string compression tool" for accomplishing this, but haven't done so yet. Cheers, Tobias On Thu, Mar 3, 2011 at 11:35 AM, Balazs E. Pataki wrote: > Hi, > > I have a big database based on Neo4J 1.2. Now, if I would like to use > the "short strings" feature of Neo4j 1.3 M03 I should regenerate my full > database, that is all strings should be reset so that it may or may not > be stored according to the new "short strings" policy. > > It seems to me that the easiest way to do this would be to somehow be > able to copy the full 1.2 database to a newly created 1.3 M03 database > by traversing the 1.2 database. But there maybe a simpler (neo4j > builtin) way to do this. Any hints about this? > > Thanks, > --- > balazs > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Slow down on insertion as db grow
Hi Massimo, I'm terribly sorry, I've had a lot of things on my plate, and this sadly fell off. Thanks for the reminder! Could you please try lowering the heap size (to somewhere between 512M-1G with your available RAM). The Neo4j memory mapping is managed outside of the Java heap. What you want to achieve for ideal performance is to get the entire nodestore and relationshipstore to fit in the memory mapped file cache. You might have to tweak the memory mapping settings to make sure that they are configured to prefer the nodestore and relationship, and have them fully memory mapped. See documentation here: http://docs.neo4j.org/chunked/snapshot/configuration-caches.html#_configuration I didn't see any information on which operating system you are using, so I assumed linux or some other unix. If you are running on Windows the memory mapping IS managed in the Java heap, then you should keep your heap settings at the same size (1.5G) and instead just focus on configuring the memory mapping so that the nodestore and relationship store are fully memory mapped. Cheers, Tobias On Thu, Mar 3, 2011 at 11:18 AM, Massimo Lusetti wrote: > On Fri, Feb 18, 2011 at 2:31 PM, Tobias Ivarsson > wrote: > > > WRT the slowdown you've seen, I'll have to investigate that further. > > Didn't mean to bother anyone but do you have any news regarding the > slowdown I'm still experiencing... !? > > Cheers > -- > Massimo > http://meridio.blogspot.com > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] ServerPlugin example GetAll failed to load
Looks like we should augment that exception to include some information about what the name was that it didn't except :) -tobias On Sun, Feb 27, 2011 at 6:49 PM, Andreas Kollegger < andreas.kolleg...@neotechnology.com> wrote: > Hi again, > > So, it looks like the verifyName() complaint is about the name containing > characters that aren't URL encoded. > > And, sorry, you asked for the source to ServerPlugin.java. That's here: > > https://github.com/neo4j/server/blob/master/server-api/src/main/java/org/neo4j/server/plugins/ServerPlugin.java > > Cheers, > Andreas > > On Feb 27, 2011, at 4:27 PM, Brendan Cheng wrote: > > > Hi, > > > > I tried to load the GetAll example into ServerPlugin directory of > > 1.3.M03 but found: > > > > 22:58:28 | 850 [WrapperSimpleAppMain] WARN > > org.neo4j.server.plugins.PluginManager - Failed to load plugin: > > ServerPlugin[GetAll] > > INFO | jvm 1| 2011/02/27 22:58:28 | > > java.lang.IllegalArgumentException: Name contains illegal characters > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > org.neo4j.server.plugins.ServerPlugin.verifyName(ServerPlugin.java:120) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > org.neo4j.server.plugins.PluginPoint.(PluginPoint.java:36) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > org.neo4j.server.plugins.PluginMethod.(PluginMethod.java:40) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > > org.neo4j.server.plugins.PluginPointFactoryImpl.createFrom(PluginPointFactoryImpl.java:98) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > > org.neo4j.server.plugins.ServerPlugin.getDefaultExtensionPoints(ServerPlugin.java:175) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > > org.neo4j.server.plugins.ServerPlugin.loadServerExtender(ServerPlugin.java:152) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > org.neo4j.server.plugins.PluginManager.(PluginManager.java:57) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > org.neo4j.server.plugins.PluginManager.(PluginManager.java:45) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > org.neo4j.server.modules.RESTApiModule.loadPlugins(RESTApiModule.java:75) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > org.neo4j.server.modules.RESTApiModule.start(RESTApiModule.java:50) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > > org.neo4j.server.NeoServerWithEmbeddedWebServer.startModules(NeoServerWithEmbeddedWebServer.java:114) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > > org.neo4j.server.NeoServerWithEmbeddedWebServer.start(NeoServerWithEmbeddedWebServer.java:83) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > org.neo4j.server.BootStrapper.start(BootStrapper.java:62) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > org.neo4j.server.BootStrapper.main(BootStrapper.java:115) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > java.lang.reflect.Method.invoke(Method.java:597) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > > org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:272) > > INFO | jvm 1| 2011/02/27 22:58:28 | at > > java.lang.Thread.run(Thread.java:662) > > > > I have changed a character from your example. > > Any idea? > > where can I locate your ServerPlugin.java file? > > Cheers, > > > > Brendan > > ___ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] MMap Error on importing large data
his at my machine.
> > >>
> > >
> > > I've just pushed the code to github. The test class is the
> TestOSMImport.
> > > Currently it skips a test if the test data is missing, and there is
> only
> > > data for two specific test cases in the code base (Billesholm and
> Malmö).
> > To
> > > get it to run the big tests, simply download denmark.osm and/or
> > croatia.osm
> > > from downloads.cloudmade.com. At the moment croatia.osm imports fine,
> at
> > > reasonable performance, but denmark.osm is the one giving the problems.
> > >
> > > Looks like the memory mapped buffer configuration needs to be tweaked.
> > >>
> > >
> > > From Johans previous answer, combined with something I read on the
> wiki,
> > it
> > > seems that the batch inserter needs different mmap settings than the
> > normal
> > > API. I read that the batch inserter uses the heap for its mmap, while
> the
> > > normal API does not. If I understand correctly, this means that when
> > using
> > > the batch inserter, we have to use smaller mmap, otherwise we might
> fill
> > the
> > > heap too soon?
> > >
> > > In any case, it seems like keeping mmap settings relatively small
> should
> > > avoid this problem, although might not lead to best performance? Have I
> > > understood correctly?
> > >
> > > On Windows heap buffers are used by default and auto configuration
> > >> will look how much heap is available. Getting out of memory exceptions
> > >> is an indication that the configuration passed in is using more memory
> > >> than available heap.
> > >>
> > >
> > > I am currently using -Xmx2048 on a 4GB ram machine, 32bit java, and the
> > > settings:
> > >
> > >static {
> > >NORMAL_CONFIG.put( "neostore.nodestore.db.mapped_memory", "50M"
> );
> > >NORMAL_CONFIG.put(
> > > "neostore.relationshipstore.db.mapped_memory", "150M" );
> > >NORMAL_CONFIG.put( "neostore.propertystore.db.mapped_memory",
> > "200M" );
> > >NORMAL_CONFIG.put(
> > > "neostore.propertystore.db.strings.mapped_memory", "300M" );
> > >NORMAL_CONFIG.put(
> > > "neostore.propertystore.db.arrays.mapped_memory", "10M" );
> > >NORMAL_CONFIG.put( "dump_configuration", "false" );
> > >}
> > >
> > >
> > > These settings do not seem to be too high, but if the normal graph
> > database
> > > service will allocate memory outside the heap, and the heap has already
> > been
> > > filled by the batch inserter, perhaps that is where the problem lies?
> > > Perhaps we do need a way of freeing memory more aggressively after the
> > batch
> > > insertion phase? Get the heap down before allowing the normal API
> access
> > to
> > > the memory?
> > >
> > > Regards, Craig
> > > ___
> > > Neo4j mailing list
> > > User@lists.neo4j.org
> > > https://lists.neo4j.org/mailman/listinfo/user
> > ___
> > Neo4j mailing list
> > User@lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
> >
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] String statistics after 1.3.M03
The tool is theoretically safe, and has never caused any problems during testing. It does however come with NO GUARANTEES, and running it is on your own risk. I'm not sure the input from those statistics would influence anything at this point, but since I haven't looked at any of these statistics of the string store WITH short strings in place it would at least be fun to look at. The information to be found in that data is: how many strings are reasonably short (could have been covered by one of the originally proposed short string encodings), but were not covered by the actual implementation. However, when implementing this optimization I found an embarrassing bug in my statistics code. Nothing that will harm your data, but rather make the statistics slightly misleading. Some sort of bonus points to anyone who finds it. I should perhaps fix that bug if the tool is to be used further. But on the other hand, not fixing it means that the statistics would show me how much the implementation misses because of my mistake in coding up the statistics tool, and I would actually find that interesting. I don't think I'll have time to investigate any statistics in quite a while. So if you do decide to run the tool and send data to me, I might not respond for a few weeks. Cheers, Tobias On Sun, Feb 27, 2011 at 6:44 PM, Massimo Lusetti wrote: > Hi neo4j developers, > I'm repopulating a neo4j db with fresh data made after M03 release, > so I guess If I could run Tobias statistics tool to gather some > info... are these interesting to you and safe for data? > > Cheers > -- > Massimo > http://meridio.blogspot.com > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Two new milestones: Neo4j 1.3.M03 and Spring Data Graph 1.0.0.M3 Released
On Sat, Feb 26, 2011 at 2:05 PM, rick.bullo...@burningskysoftware.com < rick.bullo...@burningskysoftware.com> wrote: > Also, I assume the new string storage system works with existing neo > databases? > Yes, as I have explained in my emails to this list before, and in my blog post describing this feature (linked from the release announcement), this feature is fully backwards compatible. -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Two new milestones: Neo4j 1.3.M03 and Spring Data Graph 1.0.0.M3 Released
Friends! Today the Neo4j team released milestone 3 of Neo4j 1.3. This release features some interesting kernel improvements, such as a better storage of short strings. It also improves the Neo4j graph database server. For example by making it easier to write server plugins. For more details, please see the release blog post: http://blog.neo4j.org/2011/02/neo4j-13-m03-with-short-strings.html As if that was not enough, we then took that fresh new release and included it in Spring Data Graph. The component for integrating Neo4j with Spring that we are building together with VMWare/SpringSource. Spring Data Graph version 1.0.0 was then also released in it's third milestone! This release contains soo many nice things that the best way to tell you about it is by including the original release note (forwarded below). For downloading Neo4j 1.3, milestone 3, please go to: http://neo4j.org/download For downloading Spring Data Graph, please see the official release announcement: http://www.springsource.org/node/3040 I would like to thank everyone who were involved in making this release possible, especially all of you who helped me with gathering statistics for better short string support. Please try out the new features in these releases, and continue to give us feedback. We really appreciate the mutual support in this community. Cheers, Tobias -- Forwarded message -- From: Thomas Risberg Date: Sat, Feb 26, 2011 at 12:03 AM Subject: Spring Data Graph - Neo4j Support 1.0.0.M3 Released Hi everyone, We are pleased to announce that a new milestone release (1.0.0.M3) of the Spring Data Graph 1.0 project with Neo4j support is now available. The new M3 release provides: * update to Neo4j-1.3.M03, AspectJ 1.6.11.M2 * added explicit support for detachable entities * added bean validation support for property changes * added external transaction manager support (XA and one phase commit best effort) * added xml namespace configuration * simplified Spring Java Config configuration class * added automatic numerical indexing and range queries * added full featured Neo4jTemplate * all methods introduced to entities are now generified and updated reference documentation. See the official announcement at http://www.springsource.org/node/3040 Feedback welcome! -- Thomas -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neovigator Weekend Project
Nice stuff! On Mon, Feb 21, 2011 at 6:18 AM, Max De Marzi Jr. wrote: > Guys, > > So I ran into the Ask Ken project ( http://askken.heroku.com/ ) by > Michael Aufreiter yesterdat, and though it was pretty awesome... so I > ported it to using Neo4j. > > Check it out: http://neovigator.heroku.com/ > > On github at https://github.com/maxdemarzi/neovigator > > Regards, > Max > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Traversal Framework question
Marko, I don't understand what you are trying to say.
Is the question about finding, given a person node, the co-creators for that
person on any project? Or is it about reasoning in the abstract co-creator
graph. I was under the impression that finding co-creators was the only
thing we were interested in, in that case path-uniqueness will do the trick.
What node-path-uniqueness guarantees is that within each path the same node
occurs at most once. In a path from a creator to her co-creator that would
mean that the creator cannot be her own co-creator. If the starting person
has created both x and y, the path through x and the path through y will be
different paths, which means that the path uniqueness will allow the same
persons to be co-creator on both projects.
Path uniqueness would get you into problem if person A created person B and
person B (who was created) was also a creator of herself, then person B
would not be discovered as a co-creator. I cannot imagine a case where that
would be sensible. It is also impossible to store in Neo4j, since
self-relationships are not allowed.
You can do something like .filter("from two steps ago") by looking two steps
back in the path of what you've traversed so far in the evaluator.
I fear that if we continue further on the path where this discussion is
currently headed, it will become too theoretical to be of use to anyone.
Let's keep such discussions isolated.
Cheers,
Tobias
On Sun, Feb 20, 2011 at 7:14 PM, Marko Rodriguez wrote:
> Hi,
>
> Just to be picky:
>
> >> The easiest way to do that in this case is by adding:
> >> .uniqueness(Uniqueness.NODE_PATH)
>
> A co-creator's co-creator can be you. Thus, marko's co-creator's co-creator
> is marko (amongst other people). In this case, unique on a path would not
> fail, no? Can you do something like .filter("from two steps ago")?
>
> Thanks,
> Marko.
>
>
>
>
> _______
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Traversal Framework question
On Sun, Feb 20, 2011 at 7:04 PM, Marko Rodriguez wrote: > Hi, > > > Traversal.description() > > .depthFirst() > > .relationships(RelationshipTypes.CREATED, Direction.BOTH) > > .traverse(developer).nodes() > > To be clear, a co-creator is someone is who has created the same things as > you and who is not you. Thus, you need to go outgoing CREATED, then incoming > CREATED, then you need to make sure that the vertex you land at is not the > one you left from -- thus, you need to filter the originating vertex. > The easiest way to do that in this case is by adding: .uniqueness(Uniqueness.NODE_PATH) Cheers, -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Online backup works in full but not incremental
On Sun, Feb 20, 2011 at 6:11 PM, Brendan wrote: > Hi, > > After I install the neo4j on ubuntu server I'm able to backup the full > database, even repeatedly but it crashed on incremental. > Could you please provide a stacktrace, and other kinds of error output from this crash. Thank you, -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Better support for large property data
ne for LargeStoreString and yet
another for SmallStoreString, to preserve backwards compatibility without
the need for a migration tool. Writing new strings would the of course only
write LargeStore string and ShortStore string, but when reading all three
types would be possible. The drawback of this would be that memory mapping
would be spread over four files instead of only two.
Keep the comments coming.
Cheers,
Tobias
-Original Message-
> From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org]
> On
> Behalf Of Tobias Ivarsson
> Sent: Friday, February 18, 2011 9:20 AM
> To: Neo user discussions
> Subject: [Neo4j] Better support for large property data
>
> Having tackled short strings, I feel up for taking a stab at long strings,
> and large binary data objects.
>
> I know that Rick Bullotta is really interested in this, and I can imagine
> others wanting to store large properties as well. I would love to get your
> input on the ideas I have, as well as hearing about the ideas you might
> have.
>
> The way I see it there are two different kinds of large data objects.
>
> The first one is long strings, or text. Imagine building a blog engine on
> Neo4j, the text body of a blog post is likely going to be around a thousand
> characters. That is a lot of blocks in the DynamicStringStore. But you
> still
> want to support shorter strings (the title of the post for example),
> without
> much overhead, so you don't want to increase the block size for the
> DynamicStringStore. In your code you want to deal with these values as
> String objects though, you don't want a different object type just because
> the string happens to be longer.
>
> The second one is large binary data objects. Data objects that are too
> large
> to want to have allocated as a String object, or even as a byte[] object.
> You want to manipulate them through some sort of streaming interface. These
> data objects are also so large that you would prefer if their content
> wasn't
> written to the transaction logs, because that would mean that Neo4j needed
> to rotate the log extremely frequently, and since you keep the logical logs
> for HA and backup, it would fill up your disks twice as quickly as it
> needed. Properties like this would, for example, be used for storing images
> that are included in the blog posts.
>
>
> For long Strings (the first point), the solution I'm thinking of is to
> replace the stringstore and arraystore with a smallstore and a largestore.
> Both being dynamic block stores as they are today, but with different block
> sizes. Then store both arrays and strings in both of these stores. The type
> of the data stored in the block is stored in the property record for the
> property that references the blocks anyhow, so there isn't a great
> advantage
> of having different block stores for strings and arrays.
>
> For BLOBs (the second point), we need additions to the API, since you want
> to work with these things in a streaming fashion.
> I am thinking that we use java.nio.channels.ReadableByteChannel for these
> properties. Why ReadableByteChannel you ask? Why not InputStream?
> First reason: InputStream can be converted to ReadableByteChannel, and vice
> versa:
>
> http://download.oracle.com/javase/6/docs/api/index.html?java/nio/channels/Ch
> annels.html<http://download.oracle.com/javase/6/docs/api/index.html?java/nio/channels/Channels.html>
> Second reason: ReadableByteChannel is a really simple interface (only three
> methods) if you want to write your own custom implementation.
>
> Setting a BLOB property would then look like this:
>
> ReadableByteChannel myBlob = ...
> node.setProperty("a_blob", myBlob);
>
> Getting would look like this:
>
> ReadableByteChannel myBlob =
> (ReadableByteChannel)node.getProperty("a_blob");
>
>
> Perhaps we could then, also come up with some nice API for appending to a
> BLOB property:
>
> ReadableByteChannel moreData = ...
> ReadableByteChannel myBlob =
> (ReadableByteChannel)node.getProperty("a_blob");
> node.setProperty( "a_blob", BlobUtils.append(myBlob, moreData) );
>
>
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Traversal Framework question
On Sun, Feb 20, 2011 at 10:48 AM, Alfredas Chmieliauskas < al.fre...@gmail.com> wrote: > Dear all, > > could somebody point me to more documentation on the new traversal > framework (besides http://wiki.neo4j.org/content/Traversal_Framework)? > Also the new Evaluator and how to use it? > That page, along with the examples pages is the best there is: * http://components.neo4j.org/neo4j-examples/snapshot/traversal.html * http://wiki.neo4j.org/content/Traversal_HowTo > > If we have a graph described in the pipes Co-Developers example > (https://github.com/tinkerpop/pipes/wiki/Using-Pipes-to-Traverse-Graphs). > Would this traversal return the co-developers? > > Traversal.description() > .depthFirst() > .relationships(RelationshipTypes.CREATED, Direction.OUTGOING) > .relationships(RelationshipTypes.CREATED, Direction.INCOMING) > .traverse(developer).nodes() > No, unfortunately it wouldn't. Relationship type specifications in TraversalDescriptions are not ordered, what you have written is just a different spelling of: Traversal.description() .depthFirst() .relationships(RelationshipTypes.CREATED, Direction.BOTH) .traverse(developer).nodes() Cheers, -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Better support for large property data
Having tackled short strings, I feel up for taking a stab at long strings,
and large binary data objects.
I know that Rick Bullotta is really interested in this, and I can imagine
others wanting to store large properties as well. I would love to get your
input on the ideas I have, as well as hearing about the ideas you might
have.
The way I see it there are two different kinds of large data objects.
The first one is long strings, or text. Imagine building a blog engine on
Neo4j, the text body of a blog post is likely going to be around a thousand
characters. That is a lot of blocks in the DynamicStringStore. But you still
want to support shorter strings (the title of the post for example), without
much overhead, so you don't want to increase the block size for the
DynamicStringStore. In your code you want to deal with these values as
String objects though, you don't want a different object type just because
the string happens to be longer.
The second one is large binary data objects. Data objects that are too large
to want to have allocated as a String object, or even as a byte[] object.
You want to manipulate them through some sort of streaming interface. These
data objects are also so large that you would prefer if their content wasn't
written to the transaction logs, because that would mean that Neo4j needed
to rotate the log extremely frequently, and since you keep the logical logs
for HA and backup, it would fill up your disks twice as quickly as it
needed. Properties like this would, for example, be used for storing images
that are included in the blog posts.
For long Strings (the first point), the solution I'm thinking of is to
replace the stringstore and arraystore with a smallstore and a largestore.
Both being dynamic block stores as they are today, but with different block
sizes. Then store both arrays and strings in both of these stores. The type
of the data stored in the block is stored in the property record for the
property that references the blocks anyhow, so there isn't a great advantage
of having different block stores for strings and arrays.
For BLOBs (the second point), we need additions to the API, since you want
to work with these things in a streaming fashion.
I am thinking that we use java.nio.channels.ReadableByteChannel for these
properties. Why ReadableByteChannel you ask? Why not InputStream?
First reason: InputStream can be converted to ReadableByteChannel, and vice
versa:
http://download.oracle.com/javase/6/docs/api/index.html?java/nio/channels/Channels.html
Second reason: ReadableByteChannel is a really simple interface (only three
methods) if you want to write your own custom implementation.
Setting a BLOB property would then look like this:
ReadableByteChannel myBlob = ...
node.setProperty("a_blob", myBlob);
Getting would look like this:
ReadableByteChannel myBlob =
(ReadableByteChannel)node.getProperty("a_blob");
Perhaps we could then, also come up with some nice API for appending to a
BLOB property:
ReadableByteChannel moreData = ...
ReadableByteChannel myBlob =
(ReadableByteChannel)node.getProperty("a_blob");
node.setProperty( "a_blob", BlobUtils.append(myBlob, moreData) );
Comment please.
--
Tobias Ivarsson
Hacker, Neo Technology
www.neotechnology.com
Cellphone: +46 706 534857
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Help us make Neo4j better at handling YOUR data
Yes, I'm thinking of making that my lab project for this weekend. I'm starting a separate thread for discussing that, title: "Better support for large property data". I'd love your input Rick. -tobias On Fri, Feb 18, 2011 at 2:43 PM, Rick Bullotta < rick.bullo...@burningskysoftware.com> wrote: > Awesome! Now wanna tackle the other end of the spectrum, really big > strings > and byte arrays? ;-) > > > > -Original Message- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > On > Behalf Of Tobias Ivarsson > Sent: Friday, February 18, 2011 8:29 AM > To: Neo user discussions > Subject: Re: [Neo4j] Help us make Neo4j better at handling YOUR data > > Thank you everyone who submitted statistics, both in this thread and to me > directly! > > As of last night, this feature is committed in Neo4j trunk, and should make > it into the next milestone release, at which point the sum of you guys > string stores should be possible to store in 65% less space. > > I've written up a blog post about how the implementation works: > http://journal.thobe.org/2011/02/better-support-for-short-strings-in.html > > Thank you so much for all your help! > -- > Tobias Ivarsson > Hacker, Neo Technology > www.neotechnology.com > Cellphone: +46 706 534857 > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > > _______ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Slow down on insertion as db grow
On Thu, Feb 17, 2011 at 3:32 PM, Massimo Lusetti wrote: > I'm trying to populate the DB with Nodes with only one String > property, an IPv4 address. This is just a test case for my typical use > case and it shows that when the process starts the first 440744 nodes > are going in at a speed of 1.05ms each Node but this speed drops to > 8.75ms after having inserted 12545155 nodes. > The final DB size is: 2.9G since i tweaked the sintrg_block_size at > graphdb creation time to 60bytes instead of 120... > I recently (yesterday) committed a new feature for Neo4j that will store these kinds of short strings, making Neo4j store them without having to involve the DynamicStringStore at all. You should see a substantial speedup from using that. IPv4 addresses will always be storeable as a short string, so if all you store as properties are IPv4 addresses the DynamicStringStore wouldn't be used at all in your use case. I've written a bit about how it works on my blog: http://journal.thobe.org/2011/02/better-support-for-short-strings-in.html The feature is available in the latest snapshot build: http://neo4j.org/get?file=neo4j-1.3-SNAPSHOT-unix.tar.gz And will be released in the next milestone. WRT the slowdown you've seen, I'll have to investigate that further. Cheers, -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Help us make Neo4j better at handling YOUR data
Thank you everyone who submitted statistics, both in this thread and to me directly! As of last night, this feature is committed in Neo4j trunk, and should make it into the next milestone release, at which point the sum of you guys string stores should be possible to store in 65% less space. I've written up a blog post about how the implementation works: http://journal.thobe.org/2011/02/better-support-for-short-strings-in.html Thank you so much for all your help! -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Help with exception using BatchInserter
There should have been a cause at the end of that trace, looking something like: caused by: java.io.IOException ... That is what I'm interested in. -tobias On Tue, Feb 15, 2011 at 10:12 AM, Pablo Pareja wrote: > I already put the stack trace in the first message: > > Unable to write record[244986398] @[32583190934] > > org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112) > > org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124) > > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251) > > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224) > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614) > > org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101) > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) > org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115) > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) > > org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351) > > the missing lines are just from my own classes ending in a > BatchInserter.shutdown() method. > > Cheers, > > Pablo > > On Tue, Feb 15, 2011 at 9:06 AM, Tobias Ivarsson < > tobias.ivars...@neotechnology.com> wrote: > > > I had a second look at this, and realized that those memory mapping > > settings > > aren't used from the batch inserter. Everything is allocated in the Java > > Heap. 63G out of 64G RAM is still a bit much, but not as big of a problem > > as > > I first thought. > > > > This also means that there could be another problem behind the exception > > you > > receive. There should have been a causing exception for the one you > listed. > > Could you please provide a full stack trace? > > > > Cheers, > > Tobias > > > > On Thu, Feb 10, 2011 at 10:42 PM, Pablo Pareja wrote: > > > > > Hi all, > > > I keep getting this exception and don't know what the reason could be: > > > > > > Unable to write record[244986398] @[32583190934] > > > > > > > > > org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112) > > > > > > > > > org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124) > > > > > > > > > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251) > > > > > > > > > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224) > > > > > > > > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614) > > > > > > > > > org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101) > > > > > > > > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) > > > > > > org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115) > > > > > > > > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) > > > > > > > > > org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351) > > > > > > I'm using a 65G RAM machine and the java process is launched with > -Xmx63G > > > the conf values I pass to the batch inserter are the following: > > > > > > neostore.nodestore.db.mapped_memory=10G > > > neostore.relationshipstore.db.mapped_memory=10G > > > neostore.propertystore.db.mapped_memory=10G > > > neostore.propertystore.db.strings.mapped_memory=10G > > > neostore.propertystore.db.arrays.mapped_memory=0M > > > > > > Besides, there were still 50G free space in the disk when this > exception > > > was > > > thrown > > > > > > I'd really appreciate any help, > > > Cheers > > > > > > -- > > > Pablo Pareja Tobes > > > LinkedInhttp://www.linkedin.com/in/pabloparejatobes > > > Twitter http://www.twitter.com/pablopareja > > > > > > http://www.ohnosequences.com > > > _______ > > > Neo4j mailing list > > &
Re: [Neo4j] Help with exception using BatchInserter
I had a second look at this, and realized that those memory mapping settings aren't used from the batch inserter. Everything is allocated in the Java Heap. 63G out of 64G RAM is still a bit much, but not as big of a problem as I first thought. This also means that there could be another problem behind the exception you receive. There should have been a causing exception for the one you listed. Could you please provide a full stack trace? Cheers, Tobias On Thu, Feb 10, 2011 at 10:42 PM, Pablo Pareja wrote: > Hi all, > I keep getting this exception and don't know what the reason could be: > > Unable to write record[244986398] @[32583190934] > > org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112) > > org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124) > > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251) > > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224) > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614) > > org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101) > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) > org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115) > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) > > org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351) > > I'm using a 65G RAM machine and the java process is launched with -Xmx63G > the conf values I pass to the batch inserter are the following: > > neostore.nodestore.db.mapped_memory=10G > neostore.relationshipstore.db.mapped_memory=10G > neostore.propertystore.db.mapped_memory=10G > neostore.propertystore.db.strings.mapped_memory=10G > neostore.propertystore.db.arrays.mapped_memory=0M > > Besides, there were still 50G free space in the disk when this exception > was > thrown > > I'd really appreciate any help, > Cheers > > -- > Pablo Pareja Tobes > LinkedInhttp://www.linkedin.com/in/pabloparejatobes > Twitter http://www.twitter.com/pablopareja > > http://www.ohnosequences.com > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Does BatchInserter auto-commit before shutting down?
BatchInserter.shutdown() is synchronous, it has to be, otherwise it wouldn't be safe at all. After shutdown() has returned it guarantees that all store files have been written, and are in a consistent state. -t On Mon, Feb 14, 2011 at 4:01 PM, Pablo Pareja wrote: > Great, thanks for the information. > Just out of curiosity then, are calls to BatchInserter.shutdown() sync or > asynchronous ? > I mean, does the program flow stops in that call till every operation the > shutdown implies has been > performed? > Cheers, > > Pablo > > On Mon, Feb 14, 2011 at 3:30 PM, Tobias Ivarsson < > tobias.ivars...@neotechnology.com> wrote: > > > The batch inserter will write data to disk as needed. The memory mapped > > regions are used to speed up writes, so that the API level writes can be > > done to the memory mapped regions which can then be written in large > > sequential chunks to disk. > > > > You do not have to shut down the batch inserter for this to happen, it is > > taken care of automatically. > > > > Cheers, > > Tobias > > > > On Mon, Feb 14, 2011 at 10:53 AM, Pablo Pareja wrote: > > > > > Hi, > > > > > > I was wondering whether once the memory-mapping configuration for > > > BatchInserter is full *(for example* > > > *in the case of neostore.relationshipstore.db.mapped_memory) *the > class > > > would auto-commit these changes > > > to disk or it would just throw an exception. > > > In the latter case, what would the options be? > > > > > > In my case I have to insert a huge amount of data that cannot be > > allocated > > > in RAM. > > > First I thought the BatchInserter would deal with this kind of > situation, > > > however I keep getting exceptions whenever the > > > program reaches the point where *(I guess) *memory mapping > configuration > > is > > > not enough. > > > > > > Since I cannot use more RAM than what I have, I thought, ok so I will > > > shutdown both inserter and index service and > > > create them again each time the program have already inserted a decent > > > amount of data. > > > Then, don't know why but after a couple of tests, at some point the > > program > > > always halts throwing a really weird exception > > > saying something about a index file that exists but cannot be accessed > in > > > the merge process ?!? > > > > > > I guess I'd be doing something wrong but cannot figure out what it is. > > > Thanks in advance > > > > > > -- > > > Pablo Pareja Tobes > > > LinkedInhttp://www.linkedin.com/in/pabloparejatobes > > > Twitter http://www.twitter.com/pablopareja > > > > > > http://www.ohnosequences.com > > > ___ > > > Neo4j mailing list > > > User@lists.neo4j.org > > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > > > > > > -- > > Tobias Ivarsson > > Hacker, Neo Technology > > www.neotechnology.com > > Cellphone: +46 706 534857 > > ___ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > -- > Pablo Pareja Tobes > LinkedInhttp://www.linkedin.com/in/pabloparejatobes > Twitter http://www.twitter.com/pablopareja > > http://www.ohnosequences.com > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Does BatchInserter auto-commit before shutting down?
The batch inserter will write data to disk as needed. The memory mapped regions are used to speed up writes, so that the API level writes can be done to the memory mapped regions which can then be written in large sequential chunks to disk. You do not have to shut down the batch inserter for this to happen, it is taken care of automatically. Cheers, Tobias On Mon, Feb 14, 2011 at 10:53 AM, Pablo Pareja wrote: > Hi, > > I was wondering whether once the memory-mapping configuration for > BatchInserter is full *(for example* > *in the case of neostore.relationshipstore.db.mapped_memory) *the class > would auto-commit these changes > to disk or it would just throw an exception. > In the latter case, what would the options be? > > In my case I have to insert a huge amount of data that cannot be allocated > in RAM. > First I thought the BatchInserter would deal with this kind of situation, > however I keep getting exceptions whenever the > program reaches the point where *(I guess) *memory mapping configuration is > not enough. > > Since I cannot use more RAM than what I have, I thought, ok so I will > shutdown both inserter and index service and > create them again each time the program have already inserted a decent > amount of data. > Then, don't know why but after a couple of tests, at some point the program > always halts throwing a really weird exception > saying something about a index file that exists but cannot be accessed in > the merge process ?!? > > I guess I'd be doing something wrong but cannot figure out what it is. > Thanks in advance > > -- > Pablo Pareja Tobes > LinkedInhttp://www.linkedin.com/in/pabloparejatobes > Twitter http://www.twitter.com/pablopareja > > http://www.ohnosequences.com > _______ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Does the index commit with the transaction?
That is correct. The lucene indexes in Neo4j are tied to the same transaction life cycle. On Sun, Feb 13, 2011 at 6:18 PM, Massimo Lusetti wrote: > Hi all, > Does the inner Lucene index commit toghether with the Transaction > success()/finish() cycle? > > I mean if I start a Transaction and do like > http://wiki.neo4j.org/content/Transactions#Big_transactions I suppose > I'm guarantee the Lucene index is synched to disk as soon as the > Transaction finish(), right? > > Cheers > -- > Massimo > http://meridio.blogspot.com > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Help with exception using BatchInserter
Your problem is likely related to the fact that 10G+10G+10G+10G + 63G > 64G. Memory mapping uses memory outside of the Java heap. All memory you use, memory mapping plus Java heap must fit in the available RAM. Cheers, Tobias On Thu, Feb 10, 2011 at 10:42 PM, Pablo Pareja wrote: > Hi all, > I keep getting this exception and don't know what the reason could be: > > Unable to write record[244986398] @[32583190934] > > org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112) > > org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124) > > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251) > > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224) > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614) > > org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101) > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) > org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115) > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) > > org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351) > > I'm using a 65G RAM machine and the java process is launched with -Xmx63G > the conf values I pass to the batch inserter are the following: > > neostore.nodestore.db.mapped_memory=10G > neostore.relationshipstore.db.mapped_memory=10G > neostore.propertystore.db.mapped_memory=10G > neostore.propertystore.db.strings.mapped_memory=10G > neostore.propertystore.db.arrays.mapped_memory=0M > > Besides, there were still 50G free space in the disk when this exception > was > thrown > > I'd really appreciate any help, > Cheers > > -- > Pablo Pareja Tobes > LinkedInhttp://www.linkedin.com/in/pabloparejatobes > Twitter http://www.twitter.com/pablopareja > > http://www.ohnosequences.com > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Help us make Neo4j better at handling YOUR data
This tool does not gather statistics for string arrays, since they are not affected by the proposed patch. The tool still runs, but if there are no String properties, output will be pretty boring Cheers, Tobias On Wed, Feb 9, 2011 at 7:04 PM, Massimo Lusetti wrote: > On Fri, Feb 4, 2011 at 5:13 PM, Tobias Ivarsson > wrote: > > Friends, > > > > Please read this, your involvement will make Neo4j more efficient! > > Does this run with a db with only String[] data?! > > Cheers > -- > Massimo > http://meridio.blogspot.com > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Calculating shortest paths in a large graph
a:46) > > >>>>>>>>at > > >>>>>> > > >>>> > > >> > com.hbc.locationservices.graph.GraphManager.findCheapestPathWithDijkstra(GraphManager.java:339) > > >>>>>>>>at > > >>>>>> > > >>>> > > >> > com.hbc.locationservices.graph.GraphManager.getNodesInCheapestPath(GraphManager.java:413) > > >>>>>>>>at > > >>>>>> > > >>>> > > >> > com.hbc.locationservices.graph.GraphManager$$FastClassByCGLIB$$167175c8.invoke() > > >>>>>>>>at > > >> net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149) > > >>>>>>>>at > > >>>>>> > > >>>> > > >> > org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688) > > >>>>>>>>at > > >>>>>> > > >>>> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > > >>>>>>>>at > > >>>>>> > > >>>> > > >> > org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110) > > >>>>>>>>at > > >>>>>> > > >>>> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > > >>>>>>>>at > > >>>>>> > > >>>> > > >> > org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621) > > >>>>>>>>at > > >>>>>> > > >>>> > > >> > com.hbc.locationservices.graph.GraphManager$$EnhancerByCGLIB$$64354524.getNodesInCheapestPath() > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Regards > > >>>>>>>> From: sxk1...@hotmail.com > > >>>>>>>> To: user@lists.neo4j.org > > >>>>>>>> Date: Sun, 30 Jan 2011 07:41:00 -0800 > > >>>>>>>> Subject: Re: [Neo4j] Calculating shortest paths in a large > > >> graph > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Hello Tobias,Thanks very much for your help, to be clear I want > > >> to > > >>>> find > > >>>>>> the path that has the least cost, in this case length is the > > >>>> determinator > > >>>>>> for doing this. The total number of nodes are roughly around a > 100 > > >> but > > >>>> not > > >>>>>> much more. I am attaching my heap dump,my code for parsing the > > >> google > > >>>> earth > > >>>>>> data as well as the actual xml file I use to parse and load the > > >> data, > > >>>>>> managing the Graph and the relationships. > > >>>>>>>> My project does the following:1) Create a physical graph based > > >> on > > >>>>>> google earth with a set of nodes and paths2) Load the contents of > > >> the > > >>>> google > > >>>>>> earth file into neo4j using regular expressions to parse the xml3) > > >> Run > > >>>> the > > >>>>>> dijkstra algorithm on this graph > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Your help is much appreciated, let me know if there's anything > > >> else > > >>>> I > > >>>>>> can provide.Regards > > >>>>>>>> > > >>>>>>>>> From: tobias.ivars...@neotechnology.com > > >>>>>>>>> Date: Sun, 30 Jan 2011 16:22:42 +0100 > > >>>>>>>>> To: user@lists.neo4j.org > > >>>>>>>>> Subject: Re: [Neo4j] Calculating shortest paths in a large > > >> graph > > >>>>>>>>> > > >>>>>>>>&
Re: [Neo4j] Solve a problem using neo4j
Hi John, The High Availability implementation of Neo4j behaves in the same way. Reads in HA are done from the local store. An interesting side effect of this is that while all instances in a HA cluster will store the same data, they can each have a different dataset in cache. This can, with some consistent request routing, be used for what we call "cache sharding". -tobias On Tue, Feb 8, 2011 at 10:10 AM, yehohan...@googlemail.com < yehohan...@gmail.com> wrote: > Hello Jim, > To add to the previous question about lazy loading, how does highavailable > implementation of neo4j behave? > > John > > Sent from my Nokia phone > -Original Message- > From: Jim Webber > Sent: 08/02/2011 09:16:36 > Subject: Re: [Neo4j] Solve a problem using neo4j > > Hi Ori, > > > 1. About the iterator - if I get a large amount of results (millions) it > > cannot be held in memory. Does the iterator work on in-memory collection, > or > > there is something like JDBC fetch size? > > For the embedded case, iterators are lazy so you won't hurt yourself. For > the REST API results of traversers are built up in memory so you can hurt > yourself. > > > 2. Is there an example for what I'm looking for or can someone give > pseudo > > code example for #3? > > That's pretty domain specific. But I found that looking at the Social > Network example gave me enough understanding of the simple traverser API to > get going: > > http://blog.neo4j.org/2009/09/social-networks-in-database-using-graph.html > http://components.neo4j.org/neo4j-examples/1.2-SNAPSHOT/social-network.html > > Jim > > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Help us make Neo4j better at handling YOUR data
Damn. That one place assumes that you don't have any empty strings. I've uploaded a patched version. Same location: https://github.com/downloads/thobe/neo4j-admin-store/stringstat.jar -tobias On Fri, Feb 4, 2011 at 6:52 PM, Rick Bullotta < rick.bullo...@burningskysoftware.com> wrote: > Same here. > > -Original Message- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] > On > Behalf Of Axel Morgner > Sent: Friday, February 04, 2011 12:29 PM > To: user@lists.neo4j.org > Subject: Re: [Neo4j] Help us make Neo4j better at handling YOUR data > > Hi Tobias, > > just ran the utility, but got an exception: > > Computing character frequencies for 205895 string records > 30% > ... 40% > ... 50% > ... 60% > ... 70% > ... 80% > ... 90% > ...100% > Matching potential encodings for 205895 string records > ...Exception in thread "main" java.lang.StringIndexOutOfBoundsException: > String index out of range: 0 > at java.lang.String.charAt(String.java:694) > at org.neo4j.admin.tool.stringstat.Numerical.matches(Numerical.java:30) > at > > org.neo4j.admin.tool.stringstat.TryAssumptions.process(TryAssumptions.java:4 > 6) > at org.neo4j.admin.tool.stringstat.Main.main(Main.java:55) > > Greetings > > Axel > > > I have written a small utility that analyzes the string properties stored > by > > Neo4j and computes some statistics about them. > > If I could get as many of you to run this tool on your stores and send > those > > statistics to me as possible, that would be great. > > > > This tool is available for download here: > > https://github.com/downloads/thobe/neo4j-admin-store/stringstat.jar > > > > To run it, all you need to do is: > > java -jar stringstat.jar /path/to/your/neo4j/store/dir > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Help us make Neo4j better at handling YOUR data
I actually got an idea on what better support for large binary data would look like just 5 minutes ago. I'll try to write a prototype this weekend. I've been thinking about how to support large binary data for a few months, as one of my background trails of thought, but I just now got the epiphany on how it could be exposed to the user. Cheers, Tobias On Fri, Feb 4, 2011 at 5:29 PM, Rick Bullotta < rick.bullo...@burningskysoftware.com> wrote: > Will do, Tobias. > > Also, I'd love to see a project start soon to help Neo4J become better at > storing *large* strings/byte arrays, also. I think with some creativity, > we > could avoid the need for the entire large string/blob to be stored in the > transaction logs, which to me, is one of the big challenges. > > In any case, I'll try to get you statistics soon. > > Rick -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Help us make Neo4j better at handling YOUR data
16 = 6 bit frequencies = n: 874452 a: 802639 e: 790998 i: 597945 : 524600 r: 496695 o: 462202 s: 457249 l: 429857 d: 327564 t: 310925 .: 217419 m: 207459 @: 203896 k: 190190 g: 180166 u: 169502 v: 158119 S: 148109 h: 144968 M: 118409 H: 101946 b: 93958 c: 85548 A: 84580 E: 77119 L: 75604 y: 67204 j: 66711 K: 63279 T: 61919 p: 60935 f: 60589 B: 51351 J: 48471 ?: 44154 N: 39113 O: 35251 W: 32999 F: 32734 G: 30821 R: 30341 z: 27002 V: 25837 ?: 25434 D: 23706 C: 22801 I: 21277 ?: 19167 ?: 17891 ?: 14752 P: 13294 x: 11184 2: 11011 w: 9972 U: 9556 Y: 7486 -: 4537 Z: 4524 ?: 4496 1: 4490 4: 3119 q: 1756 0: 1379 6: 1379 9: 1379 8: 1379 ,:609 ?:342 ?:294 9107508 strings with category bitmask 0b0 2947442 strings with category bitmask0b111 80361 strings with category bitmask 0b1000 8251 strings with category bitmask 0b1 16971 strings with category bitmask 0b10001 26235 strings with category bitmask 0b100010001 20632 strings with category bitmask 0b100010011 41383 strings with category bitmask 0b101010011 86358 strings with category bitmask 0b10111 508 strings with category bitmask 0b11000 32973 strings with category bitmask 0b11001 26579 strings with category bitmask 0b110010001 = Category index = 0. NineSevenBitAscii 1. LowerCaseHexadecimal 2. UpperCaseHexadecimal 3. PunctuatedNumerical 4. AlphaNumericalName 5. Numerical 6. FrequencyBased:4bit 7. FrequencyBased:5bit 8. FrequencyBased:6bit The information I want you to send back to me is the data between the and lines. May the source be with you, -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Choosing HTTP method for server plugins (REST API)
I think it makes sense to expose the invocation of a plugin by GET in some (many/most?) cases, but I don't think it makes sense to expose this to the plugin author. REST is not going to be the only transport layer of the server for ever, and other transport layers don't make the same distinction between SET/GET/POST/DELETE. I'm happy to have a discussion about this though :-) Cheers, Tobias On Fri, Feb 4, 2011 at 4:01 PM, Maximilian Schulz wrote: > I am wondering if it is possible to set the http method for server plugins > which extend the REST API. The example code only exposes methods which > respond to POST requests. But I think it would make much more sense to use > GET in many of my (and the example) cases. > > Max > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Tobias Ivarsson Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
