Re: [Neo4j] There's already an HA cluster managed by this ZooKeeper cluster
Hi, Two questions, two requests: 1) Does this happen only on one machine or any member of the cluster? Is it reproducible at will? 1) Could you please provide the messages.log for the other two machines as well? 2) Can you please tell me the result of this scenario: - Create the cluster from scratch - Restart one cluster member - It will fail as you have described - Wipe that member's database, start it - It will successfully join the cluster, as you have described. - What happens if you restart it again? thank you, CG On Mon, Dec 5, 2011 at 2:26 PM, anirudha anir...@thoughtworks.com wrote: Link for the messages.log file : https://gist.github.com/1433340 -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/There-s-already-an-HA-cluster-managed-by-this-ZooKeeper-cluster-tp3554731p3561268.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Problem Starting the Server from Java
Ok, a shot in the dark here.
There seems to be a bug related with this error message, see
http://java.net/jira/browse/JERSEY-601
Does the code you use to startup the Neo4j server work nicely outside
of GF? If yes, then please try the workaround in the link above and
also see
http://jersey.java.net/nonav/documentation/latest/glassfish.html
which might be of use for forcing the neo4j provided jersey version to be used.
If startup fails outside of GF as well then please provide the new
stacktrace so we can have a closer look.
Hope that helps.
cheers,
CG
On Sun, Nov 27, 2011 at 8:18 PM, tequ...@tequnix.org wrote:
I start the Server (latest stable ha-version) in a EJB-Application via a
Singleton Bean with Glassfish 3.1.
I do exactly what is described here:
http://docs.neo4j.org/chunked/stable/server-embedded.html#_starting_the_server_from_java
when i try to access 127.0.0.1:7474 i get a 503 error (see stacktrace
below)
any ideas what i can do to get it work ?
thanks,
¨reinhard
HTTP ERROR 503
Problem accessing /. Reason:
java.lang.RuntimeException: javax.naming.NamingException: Lookup failed
for 'com/sun/jersey/config/CDIExtension' in
SerialContext[myEnv={java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory,
java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl,
java.naming.factory.url.pkgs=com.sun.enterprise.naming} [Root exception is
javax.naming.NameNotFoundException: CDIExtension not found]
Caused by:
javax.servlet.UnavailableException: java.lang.RuntimeException:
javax.naming.NamingException: Lookup failed for
'com/sun/jersey/config/CDIExtension' in
SerialContext[myEnv={java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory,
java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl,
java.naming.factory.url.pkgs=com.sun.enterprise.naming} [Root exception is
javax.naming.NameNotFoundException: CDIExtension not found]
at
org.mortbay.jetty.servlet.ServletHolder.makeUnavailable(ServletHolder.java:415)
at
org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:458)
at
org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:685)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at
org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.neo4j.server.web.Jetty6WebServer.startJetty(Jetty6WebServer.java:168)
at org.neo4j.server.web.Jetty6WebServer.start(Jetty6WebServer.java:105)
at
org.neo4j.server.NeoServerWithEmbeddedWebServer.startWebServer(NeoServerWithEmbeddedWebServer.java:252)
at
org.neo4j.server.NeoServerWithEmbeddedWebServer.start(NeoServerWithEmbeddedWebServer.java:106)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:87)
at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:76)
at com.callooh.recommender.db.Database.start(Database.java:48)
at com.callooh.recommender.db.ConfigDB.init(ConfigDB.java:56)
at
com.callooh.recommender.ejb.StatusMapBean.startup(StatusMapBean.java:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
com.sun.ejb.containers.interceptors.BeanCallbackInterceptor.intercept(InterceptorManager.java:1009)
at
com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:65)
at
com.sun.ejb.containers.interceptors.CallbackInvocationContext.proceed(CallbackInvocationContext.java:113)
at
com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCallback(SystemInterceptorProxy.java:138)
at
com.sun.ejb.containers.interceptors.SystemInterceptorProxy.init(SystemInterceptorProxy.java:120)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
Re: [Neo4j] Neo4j Data Capacities
Yes, this is true, with a few notes: ID reuse complicates things a bit, meaning that if you delete nodes and relationships some ids will remain unused until you restart the database. Unclean shutdowns also may require scanning of the store files to determine unused records - the Config.REBUILD_IDGENERATORS_FAST parameter. So, the 35 bit address space is an upper limit. Normally this number of lost records is minuscule and easily recoverable so not a big deal. The 36 bit address space for properties is a low limit - the id reuse issue is practically non-existent for properties and since 1.5 there is no 1-1 correspondence between property id and property entry (the smallest ratio is 1:4). It all depends on the type of property - if it classifies as a short string or short array and how big (http://docs.neo4j.org/chunked/milestone/short-strings.html). So you could have a db with around 34 billion nodes with one OUTGOING relationship per node (so up to two per node, one INCOMING and one OUTGOING, since every relationship connects two nodes) and at least 68 billion properties, with a max of 68*4=272 billion properties. That's a lot of stuff! For reference, the smallest of those files will be the node store with a size of (9 bytes/record * 2^35 records)/(2^30 bytes/gigabyte) = 288 gigabytes. So you will start hitting machine restrictions before you run out of id space. cheers, CG On Thu, Nov 24, 2011 at 2:55 PM, bm3780 bm3...@gmail.com wrote: I've read that Neo4j has data capacity limitations (http://docs.neo4j.org/chunked/milestone/capabilities-capacity.html). I would like to confirm my understandings that the node, properties, and relationships limitations are for each type (e.g. AND condition), not an either/or (e.g. OR condition). Neo4j can hold: * ~34 billions nodes, AND * ~34 billion relationships, AND * ~68 billion properties So I could theoretically have a single graph with 34 billion nodes, where each node had two properties and a single relationship. -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-Data-Capacities-tp3533552p3533552.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Sampling a Neo4j instance?
No, GraphDatabaseService wisely hides those things away. I would
suggest using instanceof and casting to EmbeddedGraphDatabase.
cheers,
CG
2011/11/16 Anders Lindström andli...@hotmail.com:
Chris, thanks again for your replies.
I realize now that I don't have the 'getConfig' method -- I'm writing a
server plugin and I only get the GraphDatabaseService interface passed to my
method, not a EmbeddedGraphDatabase. Is there an equivalent way of getting
the highest node index through the interface?
Thanks.
Date: Thu, 10 Nov 2011 12:01:31 +0200
From: chris.gio...@neotechnology.com
To: user@lists.neo4j.org
Subject: Re: [Neo4j] Sampling a Neo4j instance?
Answers inline.
2011/11/9 Anders Lindström andli...@hotmail.com:
Thanks to the both of you. I am very grateful that you took your time to
put this into code -- how's that for community!
I presume this way of getting 'highId' is constant in time? It looks
rather messy though -- is it really the most straightforward way to do it?
This is the safest way to do it, that takes into consideration crashes
and HA cluster membership.
Another way to do it is
long highId = db.getConfig().getIdGeneratorFactory().get( IdType.NODE
).getHighId();
which can return the same value with the first, if some conditions are
met. It is shorter and cast-free but i'd still use the first way.
getHighId() is a constant time operation for both ways described - it
is just a field access, with an additional long comparison for the
first case.
I am thinking about how efficient this will be. As I understand it, the
sampling misses come from deleted nodes that once was there. But if I
remember correctly, Neo4j tries to reuse these unused node indices when
new nodes are added. But is an unused node index _guaranteed_ to be used
given that there is one, or could inserting another node result in
increasing 'highId' even though some indices below it are not used?
During the lifetime of a Neo4j instance there is no id reuse for Nodes
and Relationships - deleted ids are saved however and will be reused
the next time Neo4j starts. This means that if during run A you
deleted nodes 3 and 5, the first two nodes returned by createNode() on
the next run will have ids 3 and 5 - so highId will not change.
Additionally, during run A, after deleting nodes 3 and 5, no new nodes
would have the id 3 or 5. A crash (or improper shutdown) of the
database will break this however, since the ids-to-recycle will
probably not make it to disk.
So, in short, it is guaranteed that ids *won't* be reused in the same
run but not guaranteed to be reused between runs.
My conclusion is that the sampling misses will increase with index usage
sparseness and that we will have a high rate of sampling misses when we
had many deletes and few insertions recently. Would you agree?
Yes, that is true, especially given the cost of the wasted I/O and
of handling the exception. However, this cost can go down
significantly if you keep a hash set for the ids of nodes you have
deleted and check that before asking for the node by id, instead of
catching an exception. Persisting that between runs would move you
away from encapsulated Neo4j constructs and would also be more
efficient.
Thanks again.
Regards,Anders
Date: Wed, 9 Nov 2011 19:30:36 +0200
From: chris.gio...@neotechnology.com
To: user@lists.neo4j.org
Subject: Re: [Neo4j] Sampling a Neo4j instance?
Hi,
Backing Jim's algorithm with some code:
public static void main( String[] args )
{
long SAMPLE_SIZE = 1;
EmbeddedGraphDatabase db = new EmbeddedGraphDatabase(
path/to/db/ );
// Determine the highest possible id for the node store
long highId = ( (NeoStoreXaDataSource)
db.getConfig().getTxModule().getXaDataSourceManager().getXaDataSource(
Config.DEFAULT_DATA_SOURCE_NAME )
).getNeoStore().getNodeStore().getHighId();
System.out.println( highId + is the highest id );
long i = 0;
long nextId;
// Do the sampling
Random random = new Random();
while ( i SAMPLE_SIZE )
{
nextId = Math.abs( random.nextLong() ) % highId;
try
{
db.getNodeById( nextId );
i++;
System.out.println( id + nextId + is there );
}
catch ( NotFoundException e )
{
// NotFoundException is thrown when the node asked is not
in use
System.out.println( id + nextId + not in use );
}
}
db.shutdown();
}
Like already mentioned, this will be slow. Random jumps around the
graph are not something caches can keep up with - unless your whole db
fits in memory. But accessing random pieces of an on-disk file cannot
be done much faster.
Re: [Neo4j] Sampling a Neo4j instance?
Answers inline.
2011/11/9 Anders Lindström andli...@hotmail.com:
Thanks to the both of you. I am very grateful that you took your time to put
this into code -- how's that for community!
I presume this way of getting 'highId' is constant in time? It looks rather
messy though -- is it really the most straightforward way to do it?
This is the safest way to do it, that takes into consideration crashes
and HA cluster membership.
Another way to do it is
long highId = db.getConfig().getIdGeneratorFactory().get( IdType.NODE
).getHighId();
which can return the same value with the first, if some conditions are
met. It is shorter and cast-free but i'd still use the first way.
getHighId() is a constant time operation for both ways described - it
is just a field access, with an additional long comparison for the
first case.
I am thinking about how efficient this will be. As I understand it, the
sampling misses come from deleted nodes that once was there. But if I
remember correctly, Neo4j tries to reuse these unused node indices when new
nodes are added. But is an unused node index _guaranteed_ to be used given
that there is one, or could inserting another node result in increasing
'highId' even though some indices below it are not used?
During the lifetime of a Neo4j instance there is no id reuse for Nodes
and Relationships - deleted ids are saved however and will be reused
the next time Neo4j starts. This means that if during run A you
deleted nodes 3 and 5, the first two nodes returned by createNode() on
the next run will have ids 3 and 5 - so highId will not change.
Additionally, during run A, after deleting nodes 3 and 5, no new nodes
would have the id 3 or 5. A crash (or improper shutdown) of the
database will break this however, since the ids-to-recycle will
probably not make it to disk.
So, in short, it is guaranteed that ids *won't* be reused in the same
run but not guaranteed to be reused between runs.
My conclusion is that the sampling misses will increase with index usage
sparseness and that we will have a high rate of sampling misses when we had
many deletes and few insertions recently. Would you agree?
Yes, that is true, especially given the cost of the wasted I/O and
of handling the exception. However, this cost can go down
significantly if you keep a hash set for the ids of nodes you have
deleted and check that before asking for the node by id, instead of
catching an exception. Persisting that between runs would move you
away from encapsulated Neo4j constructs and would also be more
efficient.
Thanks again.
Regards,Anders
Date: Wed, 9 Nov 2011 19:30:36 +0200
From: chris.gio...@neotechnology.com
To: user@lists.neo4j.org
Subject: Re: [Neo4j] Sampling a Neo4j instance?
Hi,
Backing Jim's algorithm with some code:
public static void main( String[] args )
{
long SAMPLE_SIZE = 1;
EmbeddedGraphDatabase db = new EmbeddedGraphDatabase(
path/to/db/ );
// Determine the highest possible id for the node store
long highId = ( (NeoStoreXaDataSource)
db.getConfig().getTxModule().getXaDataSourceManager().getXaDataSource(
Config.DEFAULT_DATA_SOURCE_NAME )
).getNeoStore().getNodeStore().getHighId();
System.out.println( highId + is the highest id );
long i = 0;
long nextId;
// Do the sampling
Random random = new Random();
while ( i SAMPLE_SIZE )
{
nextId = Math.abs( random.nextLong() ) % highId;
try
{
db.getNodeById( nextId );
i++;
System.out.println( id + nextId + is there );
}
catch ( NotFoundException e )
{
// NotFoundException is thrown when the node asked is not in
use
System.out.println( id + nextId + not in use );
}
}
db.shutdown();
}
Like already mentioned, this will be slow. Random jumps around the
graph are not something caches can keep up with - unless your whole db
fits in memory. But accessing random pieces of an on-disk file cannot
be done much faster.
cheers,
CG
On Wed, Nov 9, 2011 at 6:08 PM, Jim Webber j...@neotechnology.com wrote:
Hi Anders,
When you do getAllNodes, you're getting back an iterable so as you point
out the sample isn't random (unless it was written randomly to disk). If
you're prepared to take a scattergun approach and tolerate being
disk-bound, then you can ask for getNodeById using a made-up ID and deal
with the times when your ID's don't resolve.
It'll be slow (since the chances of having the nodes in cache are low) but
as random as your random ID generator.
Jim
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
___
Re: [Neo4j] Sampling a Neo4j instance?
Hi,
Backing Jim's algorithm with some code:
public static void main( String[] args )
{
long SAMPLE_SIZE = 1;
EmbeddedGraphDatabase db = new EmbeddedGraphDatabase(
path/to/db/ );
// Determine the highest possible id for the node store
long highId = ( (NeoStoreXaDataSource)
db.getConfig().getTxModule().getXaDataSourceManager().getXaDataSource(
Config.DEFAULT_DATA_SOURCE_NAME )
).getNeoStore().getNodeStore().getHighId();
System.out.println( highId + is the highest id );
long i = 0;
long nextId;
// Do the sampling
Random random = new Random();
while ( i SAMPLE_SIZE )
{
nextId = Math.abs( random.nextLong() ) % highId;
try
{
db.getNodeById( nextId );
i++;
System.out.println( id + nextId + is there );
}
catch ( NotFoundException e )
{
// NotFoundException is thrown when the node asked is not in use
System.out.println( id + nextId + not in use );
}
}
db.shutdown();
}
Like already mentioned, this will be slow. Random jumps around the
graph are not something caches can keep up with - unless your whole db
fits in memory. But accessing random pieces of an on-disk file cannot
be done much faster.
cheers,
CG
On Wed, Nov 9, 2011 at 6:08 PM, Jim Webber j...@neotechnology.com wrote:
Hi Anders,
When you do getAllNodes, you're getting back an iterable so as you point out
the sample isn't random (unless it was written randomly to disk). If you're
prepared to take a scattergun approach and tolerate being disk-bound, then
you can ask for getNodeById using a made-up ID and deal with the times when
your ID's don't resolve.
It'll be slow (since the chances of having the nodes in cache are low) but as
random as your random ID generator.
Jim
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] 1.5.M02 - JVM BootStrapper Issues
This issue was unintentionally introduced along with some changes needed for better control and output from the windows wrapper. It was a regression that has now been fixed. I would suggest either - waiting for the next release, or - if you are feeling adventurous, download a snapshot windows build (http://neo4j.org/download/), extract bin/windows-service-wrapper-3-SNAPSHOT.jar, rename that to windows-service-wrapper-2.jar and replace the existing jar under bin/ in your current installation - the important thing is for the new jar to have the same filename as the old one. That should fix all issues that you have. As a side note, i would really appreciate if you confirmed that your configuration runs with a SNAPSHOT build. thanks, CG On Sat, Nov 5, 2011 at 8:33 AM, Peter Neubauer peter.neuba...@neotechnology.com wrote: Romiko, this is probably linked to the same issue, https://github.com/neo4j/packaging/issues/2 Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - NOSQL for the Enterprise. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. On Thu, Nov 3, 2011 at 4:07 PM, Romiko Derbynew romiko.derby...@readify.net wrote: Hi Guys, I have downloaded the latest Community edition and notice these three custom lines of code causing the Neo4J server not to start up: wrapper.java.additional.1=-d64 wrapper.java.additional.1=-server wrapper.java.additional.1=-Xss2048k With the previous version, they worked fine. They live at the bottom of the neo4j-wrapper.conf Error I get is. java.lang.IllegalArgumentException at java.lang.ProcessImpl.init(Unknown Source) at java.lang.ProcessImpl.start(Unknown Source) at java.lang.ProcessBuilder.start(Unknown Source) at java.lang.Runtime.exec(Unknown Source) at org.neo4j.wrapper.ServerProcessConsole.doStart(ServerProcessConsole.j ava:39) at org.neo4j.wrapper.ServerProcess.init(ServerProcess.java:116) at org.neo4j.wrapper.ServerProcessConsole.init(ServerProcessConsole.ja va:29) at org.neo4j.wrapper.NeoServiceWrapper.launchAsConsoleApp(NeoServiceWrap per.java:48) at org.neo4j.wrapper.NeoServiceWrapper.main(NeoServiceWrapper.java:35) Is this expected, since I am sure the JVM should support adding these additional parameters? I want these in here, so I can always ensure it is 64bit, server mode with a stack size of 2048k ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] indexquery 12times faster than cypher -- neostore-upgrade-question
Hi, it is hard to make predictions on how long it will take to migrate a store to 1.5. What the migration time actually depends on is node and relationship population, number of properties and number of dynamic records in use. This correlates to file size but one cannot determine the time based on just that, since not in use records still take up space but do not contribute to the migration time. Also indices are not touched, so they have nothing to do with upgrade time. Based on our runs, a store of the size you mention should take in the order of 1-2 hours to upgrade, depending of course on OS, filesystem, hardware etc. As far as the safety of your data goes: It is highly recommended to do a backup of your data in a safe location before starting the upgrade process and after shutting down cleanly the neo4j database. However, we have taken all the measures we could to make sure that the process is as safe as possible and we also keep a copy of the existing database files under database_dir/upgrade_backup just to be extra cautious. In the unlikely case anything goes south, you can always fix the problem, restore the old database and try the upgrade again. hope that helps, CG On Wed, Oct 12, 2011 at 9:30 AM, st.pa st...@web.de wrote: Hi, this is great, I was still holding my breath and at first didnt even notice, then I got the snapshot and changed classpaths in my project to use 1.5.m, when I noticed the following error org.neo4j.graphdb.TransactionFailureException: Could not create data source [nioneodb], see nested exception for cause of error org.neo4j.kernel.impl.transaction.TxModule.registerDataSource(TxModule.java:153) org.neo4j.kernel.GraphDbInstance.start(GraphDbInstance.java:112) at org.neo4j.kernel.EmbeddedGraphDbImpl.init(EmbeddedGraphDbImpl.java:190) at org.neo4j.kernel.EmbeddedGraphDatabase.init(EmbeddedGraphDatabase.java:80) ... Caused by: org.neo4j.kernel.impl.storemigration.UpgradeNotAllowedByConfigurationException: To enable automatic upgrade, please set allow_store_upgrade in configuration properties at org.neo4j.kernel.impl.storemigration.ConfigMapUpgradeConfiguration.checkConfigurationAllowsAutomaticUpgrade(ConfigMapUpgradeConfiguration.java:41) at org.neo4j.kernel.impl.storemigration.StoreUpgrader.attemptUpgrade(StoreUpgrader.java:50) at org.neo4j.kernel.impl.nioneo.store.NeoStore.tryToUpgradeStores(NeoStore.java:127) at org.neo4j.kernel.impl.nioneo.store.NeoStore.verifyCorrectTypeDescriptorAndVersion(NeoStore.java:98) at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.loadStorage(CommonAbstractStore.java:196) at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.init(CommonAbstractStore.java:104) at org.neo4j.kernel.impl.nioneo.store.AbstractStore.init(AbstractStore.java:120) at org.neo4j.kernel.impl.nioneo.store.NeoStore.init(NeoStore.java:72) at org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.init(NeoStoreXaDataSource.java:137) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.neo4j.kernel.impl.transaction.XaDataSourceManager.create(XaDataSourceManager.java:75) at org.neo4j.kernel.impl.transaction.TxModule.registerDataSource(TxModule.java:147) ... 24 more which tells me to allow the neo-store to be updated. That spooked me so much that I didnt yet try it in earnest, focusing instead on rewriting my app to work without thread-interrupts because that was apparently messing with lucene, and trying to find the reason why the worker-threads' transactions always keep deadlocking... as much as I desired that cypher-feature with the RelationshipIndexLookup, there are work-arounds, but I'll get around to testing it sometime soon. Before I do upgrade the store, 39 MB neostore.nodestore.db 412 MB neostore.propertystore.db 4315 MB neostore.propertystore.db.arrays 1857 MB neostore.propertystore.db.strings 1090 MB neostore.relationshipstore.db plus 1.6GB of indices, any idea how long such a neostore-upgrade might take from 1.4.1. to that 1.5.snapshot? like seconds, minutes, hours, or worse? Am 02.10.2011 19:25, schrieb Andres Taylor: And now it's ready. If you are using SNAPSHOT, please update your dependencies, and check it out. http://docs.neo4j.org/chunked/snapshot/query-start.html#start-relationship-by-index-lookup Thanks again for providing feedback. It's much more fun coding when I know someone is using it. Andrés On Fri, Sep 30, 2011 at 10:37 AM, Andres Taylor andres.tay...@neotechnology.com wrote: Hey! This is something that has been planned since the very
Re: [Neo4j] Trying to use Neo4J with Atomikos transaction manager, issues with Lucene index
This appears to be a different issue, having to do with the implementation of Atomikos. From some preliminary testing it seems that during commit the transaction returned by Atomikos' TransactionManager#getTransaction() is null, which makes commiting of the property COW maps to fail, making impossible to see the changes in the properties of the node. The native Neo4j TransactionManager returns the Transaction object and so the LockReleaser#commitCows() call works correctly. I will get back to you on this. cheers, CG On Sun, Oct 9, 2011 at 9:12 AM, Thibaut Colar tco...@colar.net wrote: OK, I've modified the unit test and it shows something strange now ... assuming I'm not just too tired and messed up again :) So the previous test, corrected, passes fine, but now after the first transaction is commited, i start a second one where i do the same exact check (node by prop) see: https://bitbucket.org/tcolar/stuff/src/0c4116d5b6ba/AtomikosNeo4j/src/test/java/net/colar/atomikosNeo4j/AtomikosNeo4jTest.java#cl-41 Now in this second transaction the test fails org.neo4j.graphdb.NotFoundException: testProp property not found for NodeImpl#1. at org.neo4j.kernel.impl.core.Primitive.newPropertyNotFoundException(Primitive.java:173) at org.neo4j.kernel.impl.core.Primitive.getProperty(Primitive.java:168) at org.neo4j.kernel.impl.core.NodeProxy.getProperty(NodeProxy.java:145) at net.colar.atomikosNeo4j.AtomikosNeo4jTest.test(AtomikosNeo4jTest.java:59) I assume that should work no, am I doing something wrong ? Thanks again. On 10/8/11 10:34 PM, Thibaut Colar wrote: You are right, was just using that in this (poor) unit test I made quickly late last week, it's not used in the real code. So the unit test does work and does not have the issue I have in the real and much more complex code. I guess I'll try to bring more of it in this unit test until i break it too. One of the thing we use, is the neo4j events mechanism(to sync data to another system), so i wonder if maybe using that in the context of transactions might cause the issue, or maybe some other sort of threading issue(what it looked like when i was debugging). Anyway will try to see if i can give you a failing test and/or better info tomorrow or Monday. Thanks On 10/8/11 1:51 PM, Mattias Persson wrote: The implicit key is _id_ but that's an implementation detail and shouldn't be used to assert correctness of the index in unit tests. 2011/10/8 Thibaut Colartco...@colar.net Actually you are right, seems like id is the only one that fails ... i really thought that was implicitly created, could it be possible than somehow it should be but doesn't when using the transaction manager ? On 10/8/11 11:41 AM, Thibaut Colar wrote: Right, id is an implicit field ... works when not in transaction. If i remove that test, then I fail on the later index.get. As far as deleting the db, it's not in my test, but i removed the folder in y real code and manually before running the test. On 10/8/11 11:10 AM, Rick Bullotta wrote: If I remember correctly, neo has an implicit field on each index, and the name is either id or _id. On Oct 8, 2011, at 1:58 PM, Chris Gioran chris.gio...@neotechnology.com wrote: Hi Thibaut, I noticed the following snippet in your test case code: long id = node.getId(); node.setProperty(testProp, test); index.add(node, testProp, test); Assert.assertEquals(node.getProperty(testProp), test); // Lookup By id Assert.assertNotNull(graphDb.getNodeById(id)); // Now via Index Node found = index.get(id, id).getSingle(); Assert.assertNotNull(found); Assert.assertEquals(found.getId(), id); which leads to the first test failure with a null result returned. Here you are indexing a node with key testProp and value test but you go on to ask it from the index with key id and value its id. Since you haven't added that key/value pair in the index, rightfully you don't get any results - the test fails in my setup even with the native Neo4j transaction manager. I changed that to ask the node from the index with the proper key/value pair and the test passes - the same thing happens a bit further down where you ask the index for a node with a key/value of testProp/prop (instead of testProp/test). Moreover, it passes with both the native Transaction Manager and the Atomikos implementation. Also, you are not deleting the db for the test - this means that since you add to the index nodes with the same key/value the test will pass only on the first run as getSingle() on the index hits will find the previously added nodes and it will fail. Could you try that out and see if it solves your issues? I am glad that this functionality is being used in production - please provide any feedback on any problems and what you think of this feature. hope
Re: [Neo4j] Trying to use Neo4J with Atomikos transaction manager, issues with Lucene index
Hi Thibaut, I noticed the following snippet in your test case code: long id = node.getId(); node.setProperty(testProp, test); index.add(node, testProp, test); Assert.assertEquals(node.getProperty(testProp), test); // Lookup By id Assert.assertNotNull(graphDb.getNodeById(id)); // Now via Index Node found = index.get(id, id).getSingle(); Assert.assertNotNull(found); Assert.assertEquals(found.getId(), id); which leads to the first test failure with a null result returned. Here you are indexing a node with key testProp and value test but you go on to ask it from the index with key id and value its id. Since you haven't added that key/value pair in the index, rightfully you don't get any results - the test fails in my setup even with the native Neo4j transaction manager. I changed that to ask the node from the index with the proper key/value pair and the test passes - the same thing happens a bit further down where you ask the index for a node with a key/value of testProp/prop (instead of testProp/test). Moreover, it passes with both the native Transaction Manager and the Atomikos implementation. Also, you are not deleting the db for the test - this means that since you add to the index nodes with the same key/value the test will pass only on the first run as getSingle() on the index hits will find the previously added nodes and it will fail. Could you try that out and see if it solves your issues? I am glad that this functionality is being used in production - please provide any feedback on any problems and what you think of this feature. hope that helps, CG On Thu, Oct 6, 2011 at 7:50 PM, tcolar tco...@colar.net wrote: I've been trying to make this work for a few days but while it mostly works, the Lucene index just won't. We have a fairly complex setup including neo4j, mysql, mongo and JMS ad trying to do transactions accross all that with Atomikos. Since that is quite complex, I've made a smaller unit test just using Atomikos and Neo4j to demonstrate the issue: https://bitbucket.org/tcolar/stuff/src/ddd17191e9a4/AtomikosNeo4j The main test is here: https://bitbucket.org/tcolar/stuff/src/ddd17191e9a4/AtomikosNeo4j/src/test/java/net/colar/atomikosNeo4j/AtomikosNeo4jTest.java and results: https://bitbucket.org/tcolar/stuff/src/ddd17191e9a4/AtomikosNeo4j/test.log I used http://digitalstain.blogspot.com/2010/11/using-jotm-as-transactionmanager-in.html to get me going. And my code for that part is here: https://bitbucket.org/tcolar/stuff/src/ddd17191e9a4/AtomikosNeo4j/src/main/java/net/colar/atomikosNeo4j I'm thinking that's probably where I'm doing something wrong maybe ? I'm probably not doing something right but i can't seem to find what's going on with Lucene. The test works if not using Atomikos but just plain Neo4j Transactions. Anybody as an idea what is wrong? Thanks. -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/Trying-to-use-Neo4J-with-Atomikos-transaction-manager-issues-with-Lucene-index-tp3400319p3400319.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4j Internals
On Thu, Oct 6, 2011 at 4:27 PM, Alexandre de Assis Bento Lima as...@cos.ufrj.br wrote: Thank you, Chris! BTY, do you intend to update your blog with the new information? It seems harder to do for the older posts - it comes easier to just write up changes as they are worked on and integrated. This means by the way that the upcoming property stuff will probably get their own writeup. Stay tuned. Also, keep in mind that as Neo4j evolves, all team members write up interesting stuff they work on. Look at http://planet.neo4j.org/ for a list of posts covering various aspects of Neo4j, including, but not limited to, some very interesting walk-throughs of kernel parts. have fun, CG Best, Alexandre. On Thu, 6 Oct 2011 11:02:18 +0300, Chris Gioran wrote Hi Alexandre, first off, the overall structure of the database has not changed in any drastic way. This means that most of the code paths are the same, maybe with some classes/methods renamed, so with a bit of patience you can follow through the new code. The most dramatic changes have been in the store files layout. Bits have had their semantics changed to support a bigger id space and, moreover, 1.5 will bring a complete rewrite of the PropertyStore. However the code is again localized, so you should be able to find your way around pretty quick - in-code comments should be helpful there. If you have any trouble walking through the code I am sure you will find this list very helpful. thank you, CG On Wed, Oct 5, 2011 at 6:50 PM, Alexandre de Assis Bento Lima as...@cos.ufrj.br wrote: Hi, I couldn't find any information about Neo4j storage system and other internal details on its website. Searching the web, I found: http://nosql.mypopescu.com/post/1396301953/neo4j-internals which gives many details and seems to be very good material. My question is: as the articles were published on October, 2010, are they still up-to-date? Neo4j 1.5 has the same implementation? Thanks a lot and special thanks to Chris Gioran, who wrote the articles. Alexandre. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user Alexandre A. B. Lima. --- COPPE / Computer Science Federal University of Rio de Janeiro Rio de Janeiro, Brazil. +55-21-2562-8690 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] InvalidRecordException: Record[1983624] not in useRecord[1983624] not in use
Hi Rene,
According to git log, 1,4.M04 was tagged 10/06/2011, while the fix for
a bug in the BatchInserterImpl that caused corrupted stores which gave
the message you are seeing is
https://github.com/neo4j/community/commit/71fbe4ac1174fad16a7bb154c9f78d3f1b377248
which has a date of 04/07/2011.
This makes 1.4.M06 the earliest release to use that does not have this
issue or, even better, go for 1.4.1 which is the latest stable
release. Since you are comfortable using milestones, 1.5.M01 might be
of interest to you.
http://neo4j.org/download has the links that you need.
hope that helped,
CG
2011/9/28 René Pickhardt r.pickha...@googlemail.com:
Neo1.4.M04 community edition no self build.
2011/9/28 Mattias Persson matt...@neotechnology.com
Which version of neo4j did you use to import your data, was it a
downloadable package, from maven or building yourself from a branch?
2011/9/28 René Pickhardt r.pickha...@googlemail.com
hey everyone
I got a strange error message that a record is not in use if I call
hasRelationship() function even though I checked against null before! My
real purpose was to start the traverser but it exited with the same error
message. That is why I put the n.hasRelationship which also would not
work.
I imported a graph with 50 mio. nodes and about the same size of
relationships. (though I am about to import more relationships later on)
I
put 2 mio. nodes in a lucence index called article.
the counting variable t has the value 2385 (so the first 2385 calls work
fine) if I skip this node the same problem accoures again at t = 3239.
maybe it is usefull to say that I inserted the graph using the batch
inserter.
IndexHitsNode res = article.get(key, values[0]);
if (res!=null){
Node n = res.getSingle();
if (n!=null){
if (n.hasRelationship(DynamicRelationshipType.withName( UPDATE ),
Direction.BOTH)){
//n.traverse(Order.BREADTH_FIRST,StopEvaluator.DEPTH_ONE,
ReturnableEvaluator.ALL_BUT_START_NODE,DynamicRelationshipType.withName(
UPDATE ) );
t++;
}
}
callstack:
org.neo4j.kernel.impl.nioneo.store.InvalidRecordException:
Record[1983624]
not in useRecord[1983624] not in use
at
org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java:230)
at
org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getChainRecord(RelationshipStore.java:337)
at
org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:114)
at
org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:97)
at
org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:108)
at
org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:603)
at
org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:351)
at
org.neo4j.kernel.impl.core.NodeImpl.loadInitialRelationships(NodeImpl.java:318)
at
org.neo4j.kernel.impl.core.NodeImpl.ensureRelationshipMapNotNull(NodeImpl.java:305)
at
org.neo4j.kernel.impl.core.NodeImpl.getAllRelationshipsOfType(NodeImpl.java:156)
at
org.neo4j.kernel.impl.core.NodeImpl.getRelationships(NodeImpl.java:236)
at
org.neo4j.kernel.impl.core.NodeImpl.hasRelationship(NodeImpl.java:576)
at
org.neo4j.kernel.impl.core.NodeProxy.hasRelationship(NodeProxy.java:109)
at IndexEvaluation.GenerateStreams(IndexEvaluation.java:124)
at IndexEvaluation.init(IndexEvaluation.java:98)
at EntryPoint.main(EntryPoint.java:20)
--
--
mobile: +49 (0)176 6433 2481
Skype: +49 (0)6131 / 4958926
Skype: rene.pickhardt
www.rene-pickhardt.de
http://www.beijing-china-blog.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
--
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
--
--
mobile: +49 (0)176 6433 2481
Skype: +49 (0)6131 / 4958926
Skype: rene.pickhardt
www.rene-pickhardt.de
http://www.beijing-china-blog.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Garbage Collection not in wrapper config
Hi, if i understand you correctly, you are looking at the documentation for snapshot (currently 1.5) but looking at the configuration file from the 1st milestone of 1.4 (1.4.M01). From what git history shows, these lines was added on May 30, 2011, which places them in the distribution around 1.4M04. Anyway, i just downloaded a community snapshot (1.5-SNAPSHOT) from neo4j.org and conf/neo4j-wrapper.conf has the expected contents, lines 8-9. Is there something that i am missing here? Note that adding manually this line in the configuration in versions before 1.4M04 should work as expected. cheers, CG On Mon, Sep 19, 2011 at 8:16 AM, Romiko Derbynew romiko.derby...@readify.net wrote: Hi Guys, I notice the docs (http://docs.neo4j.org/chunked/snapshot/server-configuration.html) mention that this line of config exists commented out. However, I cannot find it in the neo4j-wrapper.conf wrapper.java.additional.3=-Xloggc:data/log/neo4j-gc.log (1.4.M01 (2011-04-28) Is the docs out of date, or should this be added back, I see it as a very useful tool to leave commented out, as I can use this with gcviewer? Romiko ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4j low-level data storage
Hi, shameless plug explanation=my blog You can find a decent explanation for pre-1.4 storage layout here: http://digitalstain.blogspot.com/2010/10/neo4j-internals-file-storage.html It has not been updated however to explain extended addressing for 4B+ entities in a db (which do not change the layout, just the semantics of some bits). Nevertheless, the basic structures (fixed size records, doubly linked list etc) are still there. For the way I/O is performed (and minimized/improved upon) you can look at the way memory mapping is used here: http://digitalstain.blogspot.com/2010/10/neo4j-internals-persistence-and-memory.html And of course, there is object caching explained here: http://digitalstain.blogspot.com/2010/10/neo4j-internals-caching.html / shameless plug The organization in general is greatly favored by SSDs which are more friendly to linked lists on disk. However, there is no optimization in place yet targeting a specific storage technology - as in most applications, the better the I/O performance, the better Neo4j does. I'd be happy to help dive into more specific questions as they come to you. cheers, CG On Wed, Sep 14, 2011 at 8:09 PM, danielb danielbercht...@gmail.com wrote: Hello everybody, I have some questions regarding data storage in neo4j. How does neo4j store the data on the physical level? Are elements which have a close relationship in the graph stored in an adjacent way on disk? Any special binary format? How do you treat SSDs compared to common harddisks? What are your measures to improve I/O? Is there a technical documention which describes this? regards, Daniel -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-low-level-data-storage-tp3336483p3336483.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Size on disk, and number of properties
Hi Assem The logical log file is where all changes to be performed to the store are written out before they are actually applied - also referred to as the Write Ahead Log. The file that hosts this is bound to a specific size above which a rotation happens: a new file is created to host the WAL, all pending transactions are moved over, the store files are flushed and the old file marked as unused. Setting the configuration option keep_logical_logs to false deletes the old log files instead of keeping them around when that size limit is hit and a rotation is triggered. Hope that clear things up. cheers, CG On Wed, Sep 7, 2011 at 8:58 PM, Aseem Kishore aseem.kish...@gmail.com wrote: Thanks Johan! Configuring Neo4j with keep_logical_logs=false the logs will automatically be deleted upon rotation. What does upon rotation mean here? Aseem On Wed, Sep 7, 2011 at 5:56 AM, Johan Svensson jo...@neotechnology.comwrote: Removing the log files ending with .vversion number at runtime is perfectly safe to do but will turn off the ability to do incremental backups. You can however still perform live full backups. Configuring Neo4j with keep_logical_logs=false the logs will automatically be deleted upon rotation. -Johan On Sat, Sep 3, 2011 at 1:49 AM, Aseem Kishore aseem.kish...@gmail.com wrote: Thanks for the insights Johan! Regarding the existing disk space then, by far the bulk of it is from the logs. Is there a way to prune or garbage collect them? Is simply deleting the files safe? Should the db be off if I do that? Etc. Thanks much! Aseem On Tue, Aug 30, 2011 at 2:47 AM, Johan Svensson jo...@neotechnology.com wrote: Hi Aseem, This is actually expected behavior when performing file copy of running db and starting up with default configuration. If you remove the files ending with .id in the db directory on the local snapshot and start up setting rebuild_idgenerators_fast=false you should see the accurate amount of nodes, relationships and properties. Regarding the amount of properties not matching this could be due to a non clean shutdown on the production system. We are planing on improving this in the near future by allowing for more aggressive reuse of ids for properties. This will specifically improve things for workloads that perform a lot of property updates. -Johan On Tue, Aug 30, 2011 at 10:05 AM, Aseem Kishore aseem.kish...@gmail.com wrote: Hey guys, We do offline backups of our db on a semi-regular basis (every few days), where we (1) stop the running db, (2) copy its data directory and (3) restart the db. A few times early on, we did running backups -- but not the proper online way -- where we simply copied the data directory while the db was still running. (We did this during times where we were confident no requests were hitting the db.) We noticed that every time we did the running backup, the number of properties the web admin reported -- and the space on disk of the db -- would jump up quite a bit. We stopped doing that recently. But even now, both these numbers have gotten quite a bit higher than we expect to, and strangely, they seem to differ highly between the running db and the copies. What could be causing all of this? Here are our current numbers: *Production* - 2,338 nodes - 4,473 rels - 114,231 props (higher than we would expect it to be, but not by an order of magnitude) - *1.39 GB!* -- this is way unexpected, particularly since our db used to be in the ~10 KB ballpark, and we certainly haven't experienced hockey stick growth yet ;) The logical log only takes up 57 KB (0%) btw. *Local snapshot* - 2,338 nodes - 4,473 rels - *2,607,892 props!!!* -- ??? - *1.37 GB!* -- equally surprisingly high, but also interesting that it's less than the production db's size. 0 KB logical logs. I looked around the wiki and searched this mailing list but didn't find much clues here. But as requested on another thread, here's the output of `ls -lh data/graph.db/`: total 1474520 -rw-r--r-- 1 aseemk staff 11B Aug 30 00:46 active_tx_log drwxr-xr-x 52 aseemk staff 1.7K Aug 30 00:46 index/ -rw-r--r-- 1 aseemk staff 343B Aug 30 00:46 index.db -rw-r--r-- 1 aseemk staff 854K Aug 30 00:46 messages.log -rw-r--r-- 1 aseemk staff 36B Aug 30 00:46 neostore -rw-r--r-- 1 aseemk staff 9B Aug 30 00:46 neostore.id -rw-r--r-- 1 aseemk staff 26K Aug 30 00:46 neostore.nodestore.db -rw-r--r-- 1 aseemk staff 9B Aug 30 00:46 neostore.nodestore.db.id -rw-r--r-- 1 aseemk staff 62M Aug 30 00:46 neostore.propertystore.db -rw-r--r-- 1 aseemk staff 133B Aug 30 00:46 neostore.propertystore.db.arrays -rw-r--r-- 1 aseemk staff 9B Aug 30 00:46 neostore.propertystore.db.arrays.id
Re: [Neo4j] Online-backup error
Hi, though the command syntax seems ok, i find it strange that you try to backup from the webadmin port (7474). Have you actually changed the configuration to enable backup from port 7474? If not, and provided you have not explicitly set the backup port to something different, try as bin/neo4j-backup -full -from single://192.168.11.88 -to /home/sometime/backup since the client defaults to 6362, which is where the backup server listens to by default. If, for some reason, you need to change the backup port, change in conf/neo4j.properties the line enable_online_backup=true to enable_online_backup=port=your_port and the command to bin/neo4j-backup -full -from single://192.168.11.88:your_port -to /home/sometime/backup and you should be fine. hope that helps, CG On Wed, Aug 31, 2011 at 3:43 PM, sometime dons...@gmail.com wrote: I use bin/neo4j-backup -full -from single://192.168.11.88:7474 -to /home/sometime/backup output from console: Performing full backup from 'single://192.168.11.88:7474' Couldn't connect to 'single://192.168.11.88:7474', org.neo4j.com.ComException: Channel has been closed strange... any idea? -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/Online-backup-error-tp3298211p3298211.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] bug in neo4j-community-1.4.1-unix.tar.gz neo4j script ?
The only circumstances under which i can reproduce a *similar* failure involve a linux client, specifically an Ubuntu 11.04 installation. In more detail: I have a neo4j shell server running remotely (on a windows machine, fwiw). Then, from a recently unpacked installation of neo4j (1.4.1) i run bin/neo4j-shell -host 192.168.0.8 I manage to connect (meaning i get back a prompt) but for every entered command i get (in this case for pwd) neo4j-sh (0)$ pwd Connection refused to host: 127.0.1.1; nested exception is: java.net.ConnectException: Connection refused: connect This is tracked back to an entry in /etc/hosts file: 127.0.1.1 my_host_name After removing that line everything works. If after trying the above you still face problems, would you mind sharing some more details about your setup? - Client/server details (at least OS versions, , java -version), the more the merrier. - Network topology - A sample console session with the error thanks, CG On Fri, Aug 26, 2011 at 7:19 PM, Francesco Ruffino f.ruff...@gmail.com wrote: Thank you Chris! I'm playing with the neo4j-shell. I'm using the one I have found in the tar.gz I have tried to run this command: [myclient]$ ./neo4j-community-1.4.1/bin/neo4j-shell -host myserver -port 1337 but nothing happens: the shell remains hanging... I have checked on the server-side [myserver] netstat -an | grep 1337 I can see the port in LISTEN and then in ESTABLISHED: tcp 0 0 :::1337 :::* LISTEN tcp 0 0 :::10.226.222.155:1337 :::10.48.14.57:58329 ESTABLISHED but nothing happens on the client side... If I use the neo4j-shell directly on the server it works perfectly. what is wrong? thank you, franz 2011/8/26 Chris Gioran chris.gio...@neotechnology.com: Hi, this is a known issue. Please follow the documentation and run from the installation root (/home/ec2-user/neo4j-community-1.4.1/ in your example) as bin/neo4j command cheers, CG On Fri, Aug 26, 2011 at 1:41 PM, Francesco Ruffino f.ruff...@gmail.com wrote: [ec2-user@xxx bin]$ pwd /home/ec2-user/neo4j-community-1.4.1/bin [ec2-user@ bin]$ sudo ./neo4j start Starting Neo4j Server...Error: missing Neo4j Library, expected at /home/ec2-user/lib doesn't resolve correctly the $NEO4J_HOME variable ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] setProperty Exception
Hi John,
maybe you are trying to add a property that has a value of String[]
with at least one of the array members being null?. That would cause
the first exception. The second exception is most probably because you
try to success() and finish() your transaction even after the
exception.
For the first issue, always make sure that you are not adding null
values, not even as members of an array.
For the second issue, make sure that you have a try/catch block around
your transactional code that does tx.success() only when everything is
in place, with the catch block doing the tx.failure(). At the finally
block, just do tx.finish() - that should do it.
In code:
Transaction tx = db.beginTx();
try {
// do tx stuff here
tx.success();
}
catch (Exception e) {
tx.failure();
}
finally {
tx.finish();
}
Hope the above help. If your code does not conform to my assumptions
please share some more details.
cheers,
CG
On Thu, Aug 25, 2011 at 5:43 AM, John Howard johnyho...@gmail.com wrote:
I am using embedded 1.4.1
I get these 2 strange exceptions almost simultaneously when try to create a
new property.
java.lang.NullPointerException
at
org.neo4j.kernel.impl.nioneo.store.DynamicArrayStore.allocateFromString(DynamicArrayStore.java:142)
at
org.neo4j.kernel.impl.nioneo.store.DynamicArrayStore.allocateRecords(DynamicArrayStore.java:369)
at
org.neo4j.kernel.impl.nioneo.store.PropertyStore.allocateArrayRecords(PropertyStore.java:466)
at
org.neo4j.kernel.impl.nioneo.store.PropertyStore.encodeValue(PropertyStore.java:535)
at
org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.nodeAddProperty(WriteTransaction.java:1314)
at
org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.nodeAddProperty(WriteTransaction.java:1741)
at
org.neo4j.kernel.impl.persistence.PersistenceManager.nodeAddProperty(PersistenceManager.java:140)
at
org.neo4j.kernel.impl.core.NodeManager.nodeAddProperty(NodeManager.java:849)
at org.neo4j.kernel.impl.core.NodeImpl.addProperty(NodeImpl.java:101)
at org.neo4j.kernel.impl.core.Primitive.setProperty(Primitive.java:411)
at org.neo4j.kernel.impl.core.NodeProxy.setProperty(NodeProxy.java:120)
at graph.createProperty(graph.java:232)
and then
org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction
at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:98)
Caused by: javax.transaction.RollbackException: Failed to commit,
transaction rolledback
at
org.neo4j.kernel.impl.transaction.TxManager.rollbackCommit(TxManager.java:811)
at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:645)
at
org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:109)
at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85)
... 40 more
TIA
John
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4j Rest Server memory configuration
Hi there, see inline for my take to answering your questions. If someone has more info please share and/or correct. On Sun, Aug 21, 2011 at 10:39 AM, mike_t t.pr...@gmx.net wrote: I installed neo4j as a service on linux. Then I configured in the neo4j-community-1.4.1/conf/neo4j-wrapper.conf file following init and max memory: # Initial Java Heap Size (in MB) wrapper.java.initmemory=512 # Maximum Java Heap Size (in MB) wrapper.java.maxmemory=768 The top output shows following: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 17408 neo4j 18 0 1188m 155m 11m S 0 5.1 2:12.76 java Should neo4j not reserverd 512m memory? What did I wrong? Asking for any amount of memory does not mean that the memory pages are actually allocated - they are reserved. The JVM marks a range of addresses as reserved and the OS obliges, but through the way virtual memory works, the RAM is not actually used. This means that the JVM thinks that it has 512M but the OS knows better. The upside of not needing to do re-sizing calculations on heap growth is still there - when the JVM decides to write to a reserved page that is not still allocated, a page fault will be raised, the kernel will intervene etc, as per usual. How can I discover If neo4j is running in java server mode? Result of java -version is: OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode). Interesting question. It seems there is no vendor independent way - the client vs server thing is vendor dependent to begin with. The closest thing that seems to work is parsing the VM name string, reachable via System.getProperty( java.vm.name ) which on my machine returns Java HotSpot(TM) 64-Bit Server VM with java -version printing java version 1.6.0_24 Java(TM) SE Runtime Environment (build 1.6.0_24-b07) Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode) The same is reported of course via jconsole. Interestingly, i could not start a client JVM. The JRE's help text talks about a -server option only and that the default is server. No -client, although the JVM starts with it in the command line. I guess this has to do with all my machines being 64-bit multicore. Hope the above helped. cheers, CG -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-Rest-Server-memory-configuration-tp3272360p3272360.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4j startup script syntax errors
Hi,
yes, the lsof utility must be present (an assumption that holds for
most modern distributions). If you have a workaround for Solaris,
please share.
Could please specify which version of neo4j gives you this error?
thanks,
CG
On Tue, Aug 16, 2011 at 8:42 PM, Vyoma k.mahesh.b...@gmail.com wrote:
Hi Andreas,
I think that worked:
diff utils utils.bak
30c30
if [[ ${line} =~ ^([^#\s][^=]+)=(.+)$ ]]; then
---
if [[ ${line} =~ ^([^#\s][^=]+)=(.+)$ ]]; then
33c33
if [[ ${key} =~ ^(.*)_([0-9]+)$ ]]; then
---
if [[ ${key} =~ ^(.*)_([0-9]+)$ ]]; then
It is not giving me those syntax errors now.
It does give me an command not found error now:
./bin/neo4j start
id: illegal option -- u
Usage: id [-ap] [user]
/home/00/ntmop/neo4j
Starting Neo4j Server...WARNING: not changing user
process [8975]... waiting for server to be ready../bin/neo4j: line 172:
lsof: command not found
grep: can't open -B1
../bin/neo4j: line 172: lsof: command not found
grep: can't open -B1
BAD.
Neo4j Server may have failed to start, please check the logs.
But this must be due to lsof not being available. I think I need to get
that installed.
Thanks!
-
KMaheshBhat.com
--
View this message in context:
http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-Neo4j-startup-script-syntax-errors-tp3136965p3259350.html
Sent from the Neo4j Community Discussions mailing list archive at Nabble.com.
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] cant use index after commit
Hi Ahmed, if you are still having trouble, could you please provide a bare bones test case that reliably reproduces the problem? cheers, CG On Sun, Aug 7, 2011 at 6:56 PM, ahmed.elsharkasy ahmed.elshark...@gmail.com wrote: Are you sure no other reason? -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/cant-use-index-after-commit-tp3233038p3233302.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j.bat - java path
Hi, that is a valid complaint. The main idea was to use the java executable that is on the Path - however that can be still the case and allow for custom executable locations with just one string substitution. Consider this added into the backlog. cheers, CG On Fri, Jul 29, 2011 at 2:04 PM, Romiko Derbynew romiko.derby...@readify.net wrote: Hi Guys, Is it possible to update the neo4j.bat file for future releases to have a global variable for java Currently it is located in two places in the file neo4j.bat. We using Neo4j in the cloud and currently we have to do string replacements at two locations: e.g. - these are my configs that work on dev fabric as I do not rely on environment paths since java and neo4j are boot strapped from blob storage and the db is on a cloud drive. INSTALL SECTION set mainclass=-DserverMainClass=org.neo4j.server.Bootstrapper set binPath=java -DworkingDir... CONSOLE SECTION set mainclass=-DserverMainClass=org.neo4j.server.Bootstrapper java -DworkingDir=%~dp0.. Perhaps it would be easier to have one location to store this value, to make it easier to use absolute paths for java. We use absolute paths due to Azure VM's. e.g. $Java_Path =e:\AzureTemp\s0\deployment(337)\res\deployment(337).Neo4j.Azure.Server.Neo4jServerHost.0\directory\Neo4jInst\jre6\bin\java.exe And then set mainclass=-DserverMainClass=org.neo4j.server.Bootstrapper set binPath=$Java_Path -DworkingDir... CONSOLE SECTION set mainclass=-DserverMainClass=org.neo4j.server.Bootstrapper $Java_Path -DworkingDir=%~dp0.. We use auto configuration in the cloud, so we always customize the configs on the fly when vm's spin up. Easier way to search and replace strings would be much appreciated. This will make it much easier for us to customize the config on the fly by just replacing $Java_Path. e.g. after boot strap in the cloud. :install set classpath=-DserverClasspath=lib/*.jar;system/lib/*.jar;plugins/*.jar;system/coordinator/lib/*.jar set mainclass=-DserverMainClass=org.neo4j.server.Bootstrapper set binPath=e:\AzureTemp\s0\deployment(337)\res\deployment(337).Neo4j.Azure.Server.Neo4jServerHost.0\directory\Neo4jInst\jre6\bin\java.exe -DworkingDir=%~dps0.. -DconfigFile=conf\neo4j-wrapper.conf %classpath% %mainclass% -jar %~dps0windows-service-wrapper-1.jar %serviceName% and :console set classpath=-DserverClasspath=lib/*.jar;system/lib/*.jar;plugins/*.jar;system/coordinator/lib/*.jar set mainclass=-DserverMainClass=org.neo4j.server.Bootstrapper e:\AzureTemp\s0\deployment(337)\res\deployment(337).Neo4j.Azure.Server.Neo4jServerHost.0\directory\Neo4jInst\jre6\bin\java.exe -DworkingDir=%~dp0.. -DconfigFile=conf\neo4j-wrapper.conf %classpath% %mainclass% -jar %~dp0windows-service-wrapper-1.jar Thanks. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] strange problem while getting a node property
Hi Jean-Sébastien,
could you please tell us:
0. Are you sure that Node#0 is the one reporting that it has the
property key package?
1. How is the graph created? Specifically, how are the properties
added? The package property is being reported as existing, so it has
to get there somehow.
2. What version of Neo4j are you using?
3. Are the operations you described part of the same or different
transactions? The same instance of the database?
It would be very helpful if you provided a small piece of code that
reproduces what you see.
cheers,
CG
On Thu, Jul 28, 2011 at 2:18 PM, Jean-Sébastien Stoffen c-...@jsnet.be wrote:
Hi,
I've this strange problem when I try to collect data from the graph with
the Java API in Groovy :
db.allNodes.each {node -
cpt=0
node.getRelationships().each {rel -
cpt++
}
println (${node} ${cpt})
println node.getPropertyKeys()
}
The iteration on each node is right working.
The iteration to count the relationships on each node is working too.
The call node.getPropertyKeys() gives me the list of the properties like
this :
[nbrel, version, maintainer, section, architecture, package, priority,
dataset, installedSize]
But,
If a call node.getProperty(package)
I receive this error :
Caught: org.neo4j.graphdb.NotFoundException: package property not found
for NodeImpl#0
And, If I set the value just before, for test like this :
node.setProperty(package, test)
println node.getProperty(package)
I get the value.
So I can't get property which was not set by the node.setProperty method.
The initial data are copied into the graph with a perl script using the
Neo4j REST interface.
Maybe I do something wrong,
I'm a newbie in both Neo4j and Groovy
Regards,
Jean-Sébastien Stoffen
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] auto indexes in REST in 1.4?
In the upper left part where you can select edition and platform, the Select Release drop down list has a Current stable version (1.4) option. Select that instead of the default Current Milestone version option and you should be fine. cheers, CG On Thu, Jul 28, 2011 at 5:32 PM, dhsieh dhsie...@yahoo.com wrote: Where is the GA version download link? I can't find it in http://neo4j.org/download. -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-auto-indexes-in-REST-in-1-4-tp3159205p3206845.html Sent from the Neo4j Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Deletion From Neo4j
Hi Ahmad, Deleting a Node (or a Relationship or a Property for that matter) does not actually remove the corresponding record - it marks it as unused. Also the store is *never* compacted. The id of the deleted nodes are stored in the corresponding .id file so that they can be reused later on (after a restart). This of course means that deleting a Node does not reduce space usage but actually increases it by writing on disk its id. RelationshipTypes are not deleted, ever, neither explicitly or when you delete all Relationships that have that type. A way to remedy this is to copy over your database when you are done with your manipulations - do your additions/deletions in a scratch db and then create a new database instance and copy over your graph - this way no holes will be created and the store will be compacted, effectively defragmenting your database. The downside of this is that your ids will be not the same any more - if, for example, you deleted node with id 3 in the initial db, then you will have a node with that id in the defragged database. cheers, CG On Tue, Jul 26, 2011 at 12:53 PM, Ahmad Bakr ahmad.b...@espace.com.eg wrote: Hi All, I have a question please and may you want to help me to figure it out, i inserted about 240 nodes, 717 properties and 418 relationships with 5 relationship types. the disk space usage was 77.3 MB (including all components of Neo4j), the problem is i have to keep a specific disk space for Neo4j thus i need to delete some nodes, the strange thing is when i deleted the exisiting nodes and their relationships and properties, the disk space usage is increased to 77.4 MB and this is weird, although the admin interface of Neo4j shows that all nodes and relationship were deleted however the relationship types still 5 :) Any help ? Thanks :) -- Ahmad Bakr Software Engineer eSpace Technologies www.espace.com.eg Mob: (+2) 010-410-2280 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4j with Tomcat...need a transaction manager
Hey Jaf, On Sat, Jul 16, 2011 at 4:23 AM, etc3 e...@nextideapartners.com wrote: Michael, If I use the default trx manager in neo4j, what connection pool driver should be used in Tomcat? The internal implementation of Neo4j's transaction manager is not currently exposed for use by 3rd party resources. This means that, while capable of coordinating a 2PC you cannot use it to do so as a standalone TM like JOTM. The work we have done is actually the reverse of that - it substitutes Neo4's TM for an externally provided one, since both our transactional resources (neo store and lucene indexes) are XA compatible. For you this means that you have to use an external TM (like Atomikos and JBossTM that you mentioned - we have also tested SpringSource's implementation and JOTM and found all four of them to work) to coordinate your XA datasources. Doing so for Tomcat however is something we have not tried yet - the blog posts Michael mentioned should cover a lot of ground nevertheless, the only thing not covered is integration with the container, the rest should work. I will try and replicate your scenario locally and see what trouble may rise but I am really interested to see how it went for you. Hope that helped a bit, CG -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Michael Hunger Sent: Friday, July 15, 2011 9:05 PM To: Neo4j user discussions Subject: Re: [Neo4j] Neo4j with Tomcat...need a transaction manager Yes they were added. The XA-TX-Manager is configurable in Neo4j (defaults to our own but if you have an external one like JOTM or Atomikos) then you can configure those to be used. Example implementation for JOTM: https://github.com/digitalstain/JOTMServiceProvider Implementation in neo4j is here: https://github.com/neo4j/community/tree/master/kernel/src/main/java/org/neo4 j/kernel/impl/transaction Cheers Michael Am 16.07.2011 um 02:43 schrieb etc3: Thanks, Micheal. The blog post from Chris was from last year; were those changes for JOTM ever added to neo4j? -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Michael Hunger Sent: Friday, July 15, 2011 8:00 PM To: Neo4j user discussions Subject: Re: [Neo4j] Neo4j with Tomcat...need a transaction manager Springsource also provides/is working on their own tomcat based XA TM. Perhaps you'd like to check that out. (http://forum.springsource.org/showthread.php?76843-JTA-for-Tc-Server; p=2583 83#post258383) Otherwise there is only JOTM left. Neo4j can integrate with those TM. See also these blog posts by our team member Chris Gioran: http://digitalstain.blogspot.com/2010/11/using-jotm-as-transactionmana ger-in .html http://digitalstain.blogspot.com/2011/02/springy-ouside-graphy-inside. html http://digitalstain.blogspot.com/2010/11/neo4j-internals-transactions- part-3 -as.html Internally it uses also 2PC with the nioneo store and lucene. Cheers Michael Am 16.07.2011 um 01:37 schrieb etc3: We plan on using neo4j as the database for our website, which will run on Tomcat. Also, we'll be using ActiveMQ for sending messages within neo4j transactions, so that leaves the question of what to use for the transaction manager. Atomikos, JBossTM comes to mind, are we missing anything obvious? Does neo4j provide anything out-of-box for transactions management? Thanks Jaf ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Auto-index fulltext?
Hi Aseem, On Sat, Jul 16, 2011 at 9:46 AM, Aseem Kishore aseem.kish...@gmail.com wrote: Is the 1.4 auto-index only exact? Or can it be configured to be a fulltext index? Yes, currently the auto-indexes are only exact, there is no straightforward way to configure them explicitly. This is a known shortcoming and will be remedied pretty soon. Another addition that is coming, by the way, is the visibility of changes of the auto index within the transaction, instead of waiting for commit time as is now. (Btw, it would be awesome if we could have two auto-indexes: one exact, one full-text. It would be great in general if all indexing could be auto. Not sure when you would ever want/need manual indexing.) Well, a lot of use cases call for manual indexing, when for example it is conditional or the use of multiple indexes is required. If, however, auto indexing covers all your needs then by all means, do just that. cheers, CG Aseem ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Auto-indexing feature request: transform
Hi Aseem, This is a use case that auto indexing may not be the best fit. The purpose of auto indexing is taking care of the common use case in the simplest possible manner, which is a property added is a property indexed. Indexing a property with a different value that is present on the primitive (that is what I understood from your description, correct if wrong) is a case that is best suited for custom indexing operations. Additionally, it is a desirable quality to have an index which follows the contents of the actual database with guaranteed consistency. I would be interested to know if there are others that do similar transformations and in what manner - maybe there is room for providing a customizing API. cheers, CG On Sat, Jul 16, 2011 at 9:48 AM, Aseem Kishore aseem.kish...@gmail.com wrote: We transform some of our properties currently before (manually) indexing them. Simple examples are lowercase (fulltext indexes supports a lowercase config that defaults to true, but exact ones don't, I was disappointed to find), but more complex ones are e.g. stripping special characters. It would be great if the auto-indexer could support this notion of transforming property values (and queries) through user-defined function, just like traversal accepts a user-defined filter function. (And, fwiw, we're users of the REST API, so ideally this functionality would be exposed over REST too.) Cheers, Aseem ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Auto-index fulltext?
No, that is not what i meant. The main idea is to provide the means to configure at least some aspects of the auto index instead of relying on the default settings only. There will still be one auto index for each primitive category. However, one feature under consideration is to actually allow an arbitrary number of auto indexes, each of which will allow for individual configuration. So, when that comes along you will have what you described (and more, actually). thanks, CG On Mon, Jul 18, 2011 at 12:29 PM, Aseem Kishore aseem.kish...@gmail.com wrote: Awesome to hear, Chris, thanks. Just to clarify/confirm then: in the future, we will be able to have *both* an exact auto-index and a fulltext auto-index side-by-side? Cheers, Aseem On Mon, Jul 18, 2011 at 3:18 AM, Chris Gioran chris.gio...@neotechnology.com wrote: Hi Aseem, On Sat, Jul 16, 2011 at 9:46 AM, Aseem Kishore aseem.kish...@gmail.com wrote: Is the 1.4 auto-index only exact? Or can it be configured to be a fulltext index? Yes, currently the auto-indexes are only exact, there is no straightforward way to configure them explicitly. This is a known shortcoming and will be remedied pretty soon. Another addition that is coming, by the way, is the visibility of changes of the auto index within the transaction, instead of waiting for commit time as is now. (Btw, it would be awesome if we could have two auto-indexes: one exact, one full-text. It would be great in general if all indexing could be auto. Not sure when you would ever want/need manual indexing.) Well, a lot of use cases call for manual indexing, when for example it is conditional or the use of multiple indexes is required. If, however, auto indexing covers all your needs then by all means, do just that. cheers, CG Aseem ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Auto-index fulltext?
On Mon, Jul 18, 2011 at 1:50 PM, Aseem Kishore aseem.kish...@gmail.com wrote: Okay, we look forward to the multiple auto-indexes feature then! There will still be one auto index for each primitive category. Can you explain what a primitive category is then I apologize for the confusion. Primitives (or primitive categories) in Neo4j are Nodes and Relationships. Named so because their in-kernel classes, NodeImpl and RelationshipImpl extend org.neo4j.kernel.impl.core.Primitive Aseem On Mon, Jul 18, 2011 at 6:15 AM, Chris Gioran chris.gio...@neotechnology.com wrote: No, that is not what i meant. The main idea is to provide the means to configure at least some aspects of the auto index instead of relying on the default settings only. There will still be one auto index for each primitive category. However, one feature under consideration is to actually allow an arbitrary number of auto indexes, each of which will allow for individual configuration. So, when that comes along you will have what you described (and more, actually). thanks, CG On Mon, Jul 18, 2011 at 12:29 PM, Aseem Kishore aseem.kish...@gmail.com wrote: Awesome to hear, Chris, thanks. Just to clarify/confirm then: in the future, we will be able to have *both* an exact auto-index and a fulltext auto-index side-by-side? Cheers, Aseem On Mon, Jul 18, 2011 at 3:18 AM, Chris Gioran chris.gio...@neotechnology.com wrote: Hi Aseem, On Sat, Jul 16, 2011 at 9:46 AM, Aseem Kishore aseem.kish...@gmail.com wrote: Is the 1.4 auto-index only exact? Or can it be configured to be a fulltext index? Yes, currently the auto-indexes are only exact, there is no straightforward way to configure them explicitly. This is a known shortcoming and will be remedied pretty soon. Another addition that is coming, by the way, is the visibility of changes of the auto index within the transaction, instead of waiting for commit time as is now. (Btw, it would be awesome if we could have two auto-indexes: one exact, one full-text. It would be great in general if all indexing could be auto. Not sure when you would ever want/need manual indexing.) Well, a lot of use cases call for manual indexing, when for example it is conditional or the use of multiple indexes is required. If, however, auto indexing covers all your needs then by all means, do just that. cheers, CG Aseem ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] missing required jta.jar for debian?
Hi Eddy, what you are missing is the JTA specification jar. You can find a copy here: http://repo1.maven.org/maven2/org/apache/geronimo/specs/geronimo-jta_1.1_spec/1.1/geronimo-jta_1.1_spec-1.1.jar Neo4j uses maven to manage its dependencies so that such necessary libraries are located and downloaded automatically. This has the additional advantage that if you have a dependency management system as well all this will be automatically taken care for you. To get back to your questions, no, JavaEE is nowhere near needed for running neo4j. For the kernel you will need, apart from JTA above, http://repo1.maven.org/maven2/org/osgi/osgi_R4_core/1.0/osgi_R4_core-1.0.jar http://repo1.maven.org/maven2/org/osgi/osgi_R4_compendium/1.0/osgi_R4_compendium-1.0.jar as well and you should be good to go. For the tests, junit is also needed. I don't know of a way to have these dependencies managed from the package managers that come with linux distros, so either a dependency manager or manual downloads are your options here. In general, the pom.xml in every project has a listing of all the dependencies you will need. cheers, CG On Tue, Jul 12, 2011 at 4:18 PM, Eddy Respondek eddy.respon...@gmail.com wrote: I'm currently getting the following error message in Debian Lenny when trying to run a class file. Exception in thread main java.lang.NoClassDefFoundError: javax/transaction/TransactionManager As far as I can tell Debian is missing a jta.jar file which is apparently apart Java EE 6 but this isn't one of Debian packages. Do I need to manually install Java EE 6 or can I get JTA.jar somewhere separately? More info javac -cp /usr/share/java/neo4j-kernel.jar:/usr/share/java/jta.jar /tmp/gpneo4jimport.java java -cp /tmp:/usr/share/java/neo4j-kernel.jar:/usr/share/java/jta.jar gpneo4jimport java version 1.6.0_22 Java(TM) SE Runtime Environment (build 1.6.0_22-b04) Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode) ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Unique Constaint on Index
Hi,
the ability to acquire locks cluster-wide exists, albeit in an ad hoc
fashion. Grabbing a write lock on the node you want to ensure is
uniquely indexed will ensure that the operations are serialized across
all cluster members.
The most simple way to get that lock currently is the (somewhat
hackish but entirely correct) removal of a non-existing property.
cheers,
CG
On Thu, Jul 7, 2011 at 5:53 PM, etc3 e...@nextideapartners.com wrote:
How do I ensure another request is not performing the same operation on
another node in the cluster?
-Original Message-
From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On
Behalf Of Marko Rodriguez
Sent: Thursday, July 07, 2011 10:35 AM
To: Neo4j user discussions
Subject: Re: [Neo4j] Unique Constaint on Index
Hi,
We are testing Neo4J and need to support unique emails across all
users. Is this possible with the current API?
You can add such a constraint when updating the indices:
if(index.get('email', address).hasNext()) {
throw new RuntimeException(There are two nodes that share the same email
address.); } else {
index.put('email', address, node);
}
Marko.
http://markorodriguez.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Announcing the latest milestone (1.4.M05)
Hi Aseem, there is currently no ability to configure the auto indexer from the REST API. It is definite that this functionality will be added but it is unclear at this time when this will happen. We would be happy as well though if this was included in 1.4 GA. cheers, CG On Wed, Jun 29, 2011 at 8:46 AM, Aseem Kishore aseem.kish...@gmail.com wrote: Awesome stuff! One question: is there any ability yet to use (i.e. configure) auto-indexing from the REST API? If not, will that be a part of 1.4 final? Cheers, Aseem On Tue, Jun 28, 2011 at 7:03 AM, Andres Taylor andres.tay...@neotechnology.com wrote: Hello graphistas, Today we’re releasing the fifth and final milestone in our 1.4 “Kiruna Stol” family. We’ve expanded our feature set to include Auto Indexing and paged traversers for the REST API, the Cypher query language has seen some major improvements and as always, we’ve squeezed more performance out of our database engine. Auto indexing -- In this milestone we have included an improved version of the auto indexing functionality that was announced shortly after 1.4.M04 and has been available in development snapshots since then. You’ll now find new configuration options for separately enabling auto indexing for node and relationship properties and for defining which properties will be indexed. For example, if you want to auto-index all node properties called “name” and all relationship properties called “since” the proper configuration is: MapString, String config = new HashMapString, String(); config.put(Config.NODE_AUTO_INDEXING, “true”); config.put(Config.NODE_KEYS_INDEXABLE, “name”); config.put(Config.RELATIONSHIP_AUTO_INDEXING, “true”); config.put(Config.RELATIONSHIP_KEYS_INDEXABLE, “since”); Passing the above to the GraphDatabaseService constructor will enable auto indexing for matching nodes and relationships, and on every commit the auto index will be updated automatically. For more details, check out the javadocs and the examples in the manual available at: http://docs.neo4j.org/chunked/1.4.M05/auto-indexing.html Paged REST Traversers After including batch operations in the previous milestone, now we also provide the option to page results of traversals via a special URI contained in Node representations and that is similar to the existing Traverser REST API. You can define the page size and lease time, having this way finer control the retrieval of traverser results from the client (and also help improve performance on the server). Cypher improvements - The newly introduced Cypher query language is growing more sophisticated and in this milestone we’ve added aggregates, ordering and limits. This allows for more expressive queries, making the following a valid statement against the cineasts database: START user=(User,login,'micha') MATCH (user)-[:FRIEND]-(friend)-[r,:RATED]-(movie) RETURN movie.title, AVG(r.stars), COUNT(*) ORDER BY AVG(r.stars) DESC, COUNT(*) DESC limit 7 To see Cypher in action, check out the screencasthttp://neo4j.vidcaster.com/U2Y/introduction-to-cypher/by our own Michael Hunger. Almost there Of course that is not all. As always, we have bug fixes, performance improvements and usability enhancements, the most prevalent of which is the removal of YAJSW as the wrapper and service installer, which should bring joy to the mac users among us. With the GA release to follow shortly, we’d like you to download, try out and provide us with your valuable feedback on these new features so that we can deliver the best possible 1.4 to you, our community. For more details and links go to the release announcement herehttp://blog.neo4j.org/2011/06/neo4j-14-m05-kiruna-stol-midsummer.html . Andrés Taylor ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Parallelism
Hi, Keep in mind that Neo4j is a fully ACID database. As such, it is safe to use in the same manner as any other ACID database, since it makes use of locking over primitives, deadlock detection, isolation between transactions etc. Having said that: Yes, the API Neo4j exposes is suitable for manipulation by concurrently running threads. Each thread will run in its own transaction - the XA standard that Neo4j implements enforces a 1-1 correspondence between transaction context and thread. Yes, the API is thread safe. Of course, objects that maintain state, such as Iterators, are still not safe to publish outside the thread. The advantage of having multiple threads is dependent on your use case. If you have lots of I/O blocking operations, read-only threads that work with the memory resident set will progress faster. Or if you have multiple cores to dedicate, it probably makes sense to have one thread per such. Or simply if you want the transaction isolation offered by working in discreet threads such as when serving client requests. Or whatever other scenario you might have where multiple threads make sense. For example, building a web site over Neo4j is a good use case for having multiple threads, one per request. Creating a bulk data import tool on the other hand is not a good case to apply parallelism - since it will be constantly writing to disk, a single thread is fast enough, adding more does not increase throughput. The reasoning behind multithreaded access to Neo4j is pretty much the same as for any other use case. Whatever you choose however is well supported. cheers, CG On Fri, Jun 17, 2011 at 3:23 PM, Norbert Tausch w...@ntausch.de wrote: Hi, is it possible to traverse on a Neo4J graph DB using the Java API in parallel threads? Is this possible within one transaction or does every thread has to use its own transaction? Is the API thread-safe concerning read-only access? Is there any advantage of concerning parallelism when using Neo4j as an embedded DB? Best regards ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4J server - java
Hi Antriani, no, besides the two ways you mention, such functionality is not currently available, with a question mark over the HA way, depending on your use case. I assume what you need is to program against a remotely running instance of Neo4j through the same Java API, kind of like what is feasible through JDBC drivers for relational stores. The brand new Cypher query language is a step in this direction, providing a way to serialize 'job descriptions that can lead to a binary protocol and finally a driver that will provide what you ask for. However, the REST API is currently the proper way to talk to a Neo4j server, recently improved by batch functionality. Does your use case have a particular reason for which the REST interface is not sufficient? cheers, CG On Fri, Jun 17, 2011 at 7:00 PM, Antriani Stylianou antrian...@gmail.com wrote: Hi, A startup question here! Can I connect from java to a neo4j server running the database without using the REST API or the Remote Server.? Something like : GraphDatabaseService graphDb = new EmbeddedGraphDatabase(localhost:7474/); Thanks, A. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4J server - java
Of course, on the other hand, if the access method per se is the problem, then Michael Hunger has this to offer: https://github.com/jexp/neo4j-java-rest-binding which is a wrapper that exposes the REST server endpoint as a GraphDatabaseService. Everything works through REST so performance remains the same but you use Java APIs. cheers, CG On Fri, Jun 17, 2011 at 7:56 PM, Chris Gioran chris.gio...@neotechnology.com wrote: Glad to help! The main overhead associated with the REST interface is the request processing. The new batch request functionality available as of the latest milestone can reduce significantly that cost, so if performance against a remote instance is your main goal and you are not hesitant about using milestones, i urge you to try it out wherever you can - it will give you a significant boost and we always welcome feedback. cheers, CG On Fri, Jun 17, 2011 at 7:36 PM, Antriani Stylianou antrian...@gmail.com wrote: Hi, That answered my question! I was just aiming for performance and thought that java api would be better than REST. a. On 6/17/11 6:34 PM, Chris Gioran wrote: Hi Antriani, no, besides the two ways you mention, such functionality is not currently available, with a question mark over the HA way, depending on your use case. I assume what you need is to program against a remotely running instance of Neo4j through the same Java API, kind of like what is feasible through JDBC drivers for relational stores. The brand new Cypher query language is a step in this direction, providing a way to serialize 'job descriptions that can lead to a binary protocol and finally a driver that will provide what you ask for. However, the REST API is currently the proper way to talk to a Neo4j server, recently improved by batch functionality. Does your use case have a particular reason for which the REST interface is not sufficient? cheers, CG On Fri, Jun 17, 2011 at 7:00 PM, Antriani Stylianou antrian...@gmail.com wrote: Hi, A startup question here! Can I connect from java to a neo4j server running the database without using the REST API or the Remote Server.? Something like : GraphDatabaseService graphDb = new EmbeddedGraphDatabase(localhost:7474/); Thanks, A. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Auto Indexing for Neo4j
Good news everyone, A request that's often come up on the mailing list is a mechanism for automatically indexing properties of nodes and relationships. As of today's SNAPSHOT, auto-indexing is part of Neo4j which means nodes and relationships can now be indexed based on convention, requiring far less effort and code from the developer's point of view. Getting hold of an automatic index is straightforward: AutoIndexerNode nodeAutoIndexer = graphDb.index().getNodeAutoIndexer(); AutoIndexNode nodeAutoIndex = nodeAutoIndexer.getAutoIndex(); Once you've got an instance of AutoIndex, you can use it as a read-only IndexNode. The AutoIndexer interface also supports runtime changes and enabling/disabling the auto indexing functionality. To support the new features, there are new Config options you can pass to the startup configuration map in EmbeddedGraphDatabase, the most important of which are: Config.NODE_AUTO_INDEXING (defaults to false) Config.RELATIONSHIP_AUTO_INDEXING (defaults to false) If set to true (independently of each other) these properties will enable auto indexing functionality and at the successful finish() of each transaction, all newly added properties on the primitives for which auto indexing is enabled will be added to a special AutoIndex (and deleted or changed properties will be updated accordingly too). There are options for fine grained control to determine properties are indexed, default behaviors and so forth. For example, by default all properties are indexed. If you want only properties name and age for Nodes and since and until for Relationships to be auto indexed, simply set the initial configuration as follows: Config.NODE_KEYS_INDEXABLE = name, age; Config.RELATIONSHIP_KEYS_INDEXABLE=since, until; For the semantics of the auto-indexing operations, constraints and more detailed examples, see the documentation available at http://docs.neo4j.org/chunked/1.4-SNAPSHOT/auto-indexing.html We're pretty excited about this feature since we think it'll make your lives as developers much more productive in a range of use-cases. If you're comfortable with using SNAPSHOT versions of Neo4j, please try it out and let us know what you think - we'd really value your feedback. If you're happier with using packaged milestones then this feature will be available from 1.4 M05 in a couple of weeks from now. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Error(s) found in manifest configuration when trying to import community/pom.xml
Hi there, the problem is the maven version you are using - Neo4j does not work well with mvn 3. The exact same steps are successful with my setup which is the same but for maven 2.2.1 I suggest you downgrade your maven install or create an environment for Neo4j that uses mvn 2, until we become maven 3 compatible. cheers, CG On Sun, Jun 12, 2011 at 4:55 AM, Super Wang wangxu...@gmail.com wrote: Hi folks, I found some warnings and errors when trying to import neo4j maven project to my eclipse, here's my steps: 1. Clone git://github.com/neo4j/community.git to local E:\git\neo4j.community 2. Go to eclipse and using Import - Exist Maven Projects and point to E:\git\neo4j.community\pom.xml 3. Let it import. 4. After done, I got two projects with Error(s) found in manifest configuration errors: neo4j-server and neo4j-sunshine, below is same part of the eclipse console log 11-6-12 上午09时29分17秒: [WARN] 11-6-12 上午09时29分17秒: [WARN] Some problems were encountered while building the effective model for org.neo4j:neo4j-sunshine:jar:1.4-SNAPSHOT 11-6-12 上午09时29分17秒: [WARN] 'parent.relativePath' points at org.neo4j.build:community-build instead of org.neo4j:parent-central, please verify your project structure @ line 3, column 11 11-6-12 上午09时29分17秒: [WARN] 11-6-12 上午09时29分17秒: [WARN] It is highly recommended to fix these problems because they threaten the stability of your build. 11-6-12 上午09时29分17秒: [WARN] 11-6-12 上午09时29分17秒: [WARN] For this reason, future Maven versions might no longer support building such malformed projects. 11-6-12 上午09时29分17秒: [WARN] 11-6-12 上午09时29分17秒: [INFO] Checking licenses... 11-6-12 上午09时29分17秒: [INFO] Using 'UTF-8' encoding to copy filtered resources. 11-6-12 上午09时29分17秒: [INFO] skip non existing resourceDirectory E:\git\neo4j.community\sunshine\src\main\resources 11-6-12 上午09时29分17秒: [INFO] skip non existing resourceDirectory E:\git\neo4j.community\sunshine\src\main\resources\META-INF 11-6-12 上午09时29分17秒: [INFO] Copying 0 resource to META-INF 11-6-12 上午09时29分17秒: [INFO] Checking for multiple versions of scala 11-6-12 上午09时29分17秒: [INFO] includes = [**/*.scala,**/*.java,] 11-6-12 上午09时29分17秒: [INFO] excludes = [] 11-6-12 上午09时29分17秒: [INFO] Nothing to compile - all classes are up to date 11-6-12 上午09时29分18秒: [INFO] Nothing to compile - all classes are up to date 11-6-12 上午09时29分18秒: [INFO] Checking for multiple versions of scala 11-6-12 上午09时29分18秒: [INFO] includes = [**/*.scala,**/*.java,] 11-6-12 上午09时29分18秒: [INFO] excludes = [] 11-6-12 上午09时29分18秒: [INFO] Nothing to compile - all classes are up to date 11-6-12 上午09时29分18秒: [ERROR] Error in manifest for org.neo4j:neo4j-sunshine:jar:1.4-SNAPSHOT : The default package '.' is not permitted by the Import-Package syntax. This can be caused by compile errors in Eclipse because Eclipse creates valid class files regardless of compile errors. The following package(s) import from the default package [org.neo4j.sunshine.javacompat] 11-6-12 上午09时29分18秒: [ERROR] Error(s) found in manifest configuration 11-6-12 上午09时29分18秒: Build errors for neo4j-sunshine; org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.felix:maven-bundle-plugin:2.1.0:manifest (bundle-manifest) on project neo4j-sunshine: Error(s) found in manifest configuration 11-6-12 上午09时29分18秒: [ERROR] Error in manifest for org.neo4j:neo4j-sunshine:jar:1.4-SNAPSHOT : The default package '.' is not permitted by the Import-Package syntax. This can be caused by compile errors in Eclipse because Eclipse creates valid class files regardless of compile errors. The following package(s) import from the default package [org.neo4j.sunshine.javacompat] 11-6-12 上午09时29分18秒: [ERROR] Error(s) found in manifest configuration Any idea about this? BTW, I'm using eclipse 3.6.2 with maven 3.0.3. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Speeding up initial import of graph
Hi Daniel, I am working currently on a tool for importing big data sets into Neo4j graphs. The main problem in such operations is that the usual index implementations are just too slow for retrieving the mapping from keys to created node ids, so a custom solution is needed, that is dependent to a varying degree on the distribution of values of the input set. While your dataset is smaller than the data sizes i deal with, i would like to use it as a test case. If you could provide somehow the actual data or something that emulates them, I would be grateful. If you want to see my approach, it is available here https://github.com/digitalstain/BigDataImport The core algorithm is an XJoin style two-level-hashing scheme with adaptable eviction strategies but it is not production ready yet, mainly from an API perspective. You can contact me directly for any details regarding this issue. cheers, CG On Thu, Jun 9, 2011 at 12:59 PM, Daniel Hepper daniel.hep...@gmail.com wrote: Hi all, I'm struggling with importing a graph with about 10m nodes and 20m relationships, with nodes having 0 to 10 relationships. Creating the nodes takes about 10 minutes, but creating the relationships is slower by several orders of magnitude. I'm using a 2.4 GHz i7 MacBookPro with 4GB RAM and conventional HDD. The graph is stored as adjacency list in a text file where each line has this form: Foo|Bar|Baz (Node Foo has relations to Bar and Baz) My current approach is to iterate over the whole file twice. In the first run, I create a node with the property name for the first entry in the line (Foo in this case) and add it to an index. In the second run, I get the start node and the end nodes from the index by name and create the relationships. My code can be found here: http://pastie.org/2041801 With my approach, the best I can achieve is 100 created relationships per second. I experimented with mapped memory settings, but without much effect. Is this the speed I can expect? Any advice on how to speed up this process? Best regards, Daniel Hepper ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Problems binding to a specific ip
Hi Max, there are two distinct problems here One is that there is no IP access control that can be configured through Neo4j. Setting the listen URIs to a specific address in the configuration does not bind the server there exclusively. This behavior has to be imposed with external means, such as a firewall. The other is a benign bug that has to do with logging the listen address - currently it uses always the java.net.InetAddress methods to find the hostname, ignoring the configuration and leading to the behavior you describe, since the JVM does not prefer the localhost interface. This will be of course addressed in subsequent SNAPSHOTs and milestone releases but it will remain the default behavior. Hope that clarifies things. cheers, CG On Wed, Jun 8, 2011 at 2:59 PM, Maximilian Schulz nam...@gmail.com wrote: Hi everyone, after weeks of experimentation, we finally migrated one of our apps features to neo4j. But now that we are about to deploy the app to our staging server, we discovered a small problem with the server configuration. We have checked the manual at http://docs.neo4j.org/chunked/1.4.M03/server-configuration.html and followed the suggested configuration, but we are not getting the expected results. What we are trying to achieve is to bind the server to localhost, so it is no longer accessible from outside. We restricted access to IPv4 in the wrapper config and set the webadmin data and manage uri to http://localhost:7474. But when starting the server it still binds to the hostname. Here is the output of the start process. Check the last line, where it states that the server is started on http://OUR_DOMAIN:7474/ 6/8/11 1:44:46 PM org.neo4j.server.database.Database INFO: Using database at /tmp/neo4j-community-1.4.M03/data/graph.db 6/8/11 1:44:47 PM org.neo4j.server.modules.DiscoveryModule INFO: Mounted discovery module at [/] 6/8/11 1:44:47 PM org.neo4j.server.modules.RESTApiModule INFO: Mounted REST API at [http://localhost:7474/db/data/] 6/8/11 1:44:47 PM org.neo4j.server.modules.ManagementApiModule INFO: Mounted management API at [http://localhost:7474/db/manage/] 6/8/11 1:44:47 PM org.neo4j.server.modules.WebAdminModule INFO: Mounted webadmin at [/webadmin] 6/8/11 1:44:47 PM org.neo4j.server.NeoServerWithEmbeddedWebServer INFO: Starting Neo Server on port [7474] 6/8/11 1:44:47 PM org.neo4j.server.web.Jetty6WebServer INFO: Mounting static content at [/webadmin] from [webadmin-html] 6/8/11 1:44:48 PM org.neo4j.server.NeoServerWithEmbeddedWebServer INFO: Server started on [http://OUR_DOMAIN:7474/] I am pretty sure that I have missed something, but I cannot figure out what. Any help is greatly appreciated! Max ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Neo4j 1.4 M01 - headed to Kiruna Stol
Hi fellow graphistas. Keeping up with our schedule, two weeks after our 1.3 GA release we present to you the first milestone towards Neo4j 1.4 “Kiruna Stol”. This iteration does not introduce any major features. Instead it takes care of some annoyances in the existing codebase, incorporating your feedback to iron out issues with ops and user interaction. So if you are comfortable with 1.3, you should be right at home with this milestone, only happier! This release also paves the way for the cool new features we are considering for Kiruna Stol. Many improvements are in our sights, including paging and streaming of REST results, investigation of a query language, revisiting the API and introduce self referencing relationships and more. This won’t stop usability and performance improvements from coming though. So, go ahead and download our latest release and read more about it at http://blog.neo4j.org/2011/04/neo4j-14-m01-announcing-kiruna-stol.html and keep those suggestions coming. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Regarding Sub grouping In Graph DB
Hi Pooja, what would qualify as a subcategory in your use case? The most straightforward thing I can come up with is a property in each node named category or something similar, that has an enumerated value (not a Java Enumeration but something treated as such) that would place the node in its category. This is one of many ways to go ahead, depending on what degrees of separation you want, flexibility etc. Could you elaborate a little on what you are trying to do? cheers, CG On Mon, Apr 25, 2011 at 9:27 PM, pooja naik npooj...@yahoo.com wrote: Hi all, I am using ne04j for a IP network resource graph in my project. I would like to know whether there is a way to divide the graphical network into sub categories in neo4j? Any help or pointers is appreciated. Thanks Pooja ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node safe in a Set
On Tue, Mar 15, 2011 at 5:17 PM, Massimo Lusetti mluse...@gmail.com wrote: It's safe to use Node in a SetNode... I don't see it implements equals() nor hashCode() Of course it does! Node is the interface implemented by NodeProxy - the objects you get back and use as Nodes. NodeProxy implement equals() and hashCode() based on the id of the Node. Using them in a CollectionNode is safe. Cheers -- Massimo http://meridio.blogspot.com cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Announcing Neo4j 1.3 “Abisko Lampa” M04
Greeting fellow graphistas, Keeping up with the release schedule, today we present the fourth milestone towards the 1.3 “Abisko Lampa” version of Neo4j. The most important feature of this release is of course the expansion of the storage layer to allow for even more entities, but also includes improvements on the testing methodology used for Windows builds and the REST interface for Indexing operations. As always, extended details are available in the announcement post, reachable at http://blog.neo4j.org/2011/03/neo4j-13-abisko-lampa-m04-size-really.html There you can read a comprehensive description of the new features, however a special mention must be made to the new store format. This milestone of Neo4j brings with it an updated store version that allows for an 8x increase in the possible number of Nodes and Relationships and 16x increase for the number of Properties. All this happens without increasing the size of the database so you can go right ahead and upgrade, certain in the knowledge that everything will be done in place. A few words of caution are in order, though. An upgrade will not happen unless you explicitly request it - just pass in the configuration parameter “allow_store_upgrade” as “true” - otherwise an exception will be thrown that advises you to do just that and your store will be left untouched. There are some additional requirements for the upgrade to succeed, such as less than 65536 RelationshipTypes stored, but if you do not satisfy those the upgrade procedure will inform you of what the exact problem is and leave your store untouched. Finally, note that after a successful upgrade your database will be unreadable by previous versions. So keep in mind that while safe, this is a one way procedure. You can find more details on the subject at http://docs.neo4j.org/chunked/milestone/deployment-upgrading.html We invite you to try out these new features and provide us with your ever valuable feedback. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] IllegalStoreVersionException - Store version is older than expected...
On Wed, Mar 9, 2011 at 5:26 PM, José Devezas joseluisdeve...@gmail.com wrote: Hello, I'm using Blueprints 0.5 to read a Neo4j graph. I've just come back to the project today, maven updated to the latest JARs and now I get this error. I've tried searching for documentation on how to set this option, but I couldn't find it yet. How do I upgrade the NeoStore to the latest version? Caused by: org.neo4j.kernel.impl.nioneo.store.IllegalStoreVersionException: Store version [NeoStore v0.9.6] is older than expected, but could be upgraded automatically if 'allow_store_upgrade' configuration parameter was set to 'true'. The changes, as Peter notes, are because of the upgrade of the store format to accommodate an even larger number of entities in a Neo4j database. This of course requires an upgrade procedure. To start the latest SNAPSHOT in upgrade mode, you should pass in the configuration parameter allow_store_upgrade as true, as the exception hints. A sample way to do this is MapString, String properties = new HashMapString, String(); properties.put(Config.ALLOW_STORE_UPGRADE, true); new EmbeddedGraphDatabase(db, properties); After upgrading the store (meaning successful startup with the above or equivalent code) then you *cannot* go back to previous versions - including 1.3M03. More details on this procedure are at http://docs.neo4j.org/chunked/snapshot/deployment-upgrading.html Note however that your store will be touched only if you satisfy all the requirements the documentation outlines AND you explicitly ask for an upgrade, making the procedure safe for your data. Reversely, to use Neo4j version 1.3 you MUST upgrade to the new format. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Neo4j, Spring Framework and 2PC
Hello graphistas of the world, trying to create a more seamless coupling between Neo4j and Spring Framework, we have now built a solution for getting these two to work with an external transaction manager. This work is based on some groundwork for pluggable TMs in the Neo4j kernel and is a step in the direction of bringing support for graph operations from a managed environment in conjunction with other XA compliant data sources, the most prominent example being JDBC connections. So, start enjoying the magic through this blog post: http://digitalstain.blogspot.com/2011/02/springy-ouside-graphy-inside.html For the impatient, the code is available at https://github.com/digitalstain/Neo4j-Spring-Integration As always, feedback and suggestions are welcome and greatly appreciated. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Fatty Stack Trace to Lift your Spirits
On Wed, Dec 22, 2010 at 6:11 PM, Marko Rodriguez okramma...@gmail.com wrote: So I bumped Blueprints 0 .4-SNAPSHOT to use 1.2M06 and during the test suite evaluation, I was getting fatty stack traces being thrown all up in my face left, right, and yes, center. As long as they are not coming from below, there is no reason for alarm. However... My random thought on the matter---Luca and I have been having *potentially* similar problems with OrientDB and shutting down, opening, and deleting the DB really fast. Don't know if this is the same issue now showing up in Neo4j? The TestSuite will, in general, create a graph (directory), use it, close it, open it, delete the directory, create a new graph (directory), etc. Is the OS not having happy good time? Why happy good time with 1.2M05? I slapped up a loop doing pretty much that - create a db, store+index a few things, close, [open, read, close] (in a loop), delete, do over, for around 50 iterations. Using straight up (no tinkerpop stack) vanilla 1.2M06 on Ubuntu 10.10. No problems so far. I will leave it to run for a while, to teach my HDD some manners and I will check it again later. That lock file left behind, however, smells like crash/improper shutdown (or overlapping instances?). That really fast open/shutdown cycle you speak of, how do you do it, shell script or from main()? Does the problem appear in every cycle or intermittently? cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How to delete a node when it's already deleted ?
On Tue, Dec 7, 2010 at 3:09 PM, Andreas Ronge andreas.ro...@gmail.com wrote: An Node#isDeleted() method would also be fine. The way I see it, there are two concerns here. The first is focused at the lower levels, where the WriteTransaction/LockReleaser discover an illegal operation - deletion of an already deleted primitive. This is a hard error and at their level it should throw an exception and of course set the tx to rollback only. This is mainly an engineering decision. The second is the user level where either the same logic should apply or a check should be made first to make sure things don't go downhill. Obviously the current approach is the former. Having just a isDeleted() method is kind of awkward because it would litter the code with if statements and things would be even worse with (checked) exceptions. Maybe stealing a bit off the id would be a better solution and have the NodeImpl/NodeProxy objects do the check internally. BTW, I think that from a user perspective with the current kernel such a wrapper object (with a boolean field possibly) would be the best approach, minimizing the bookkeeping in business logic code. What I find more interesting to discuss are the semantics of operations on primitives. At the moment there is no standard to adhere to and in that respect there is a decision to be made. What I mean is: what is the proper thing to do, conceptually, when doing basic primitive manipulations. Since there is an effort to standardize a graph traversal algebra, a similar thing should be done on a data definition level, with rationalization and detailed description of what is the Right Thing (TM) to do when, for instance, one deletes a Node from a graph, regardless of implementation. Obviously my thinking is influenced from the relational model, where there are hard constraints on different things - primary keys are an obvious example here. In that case, the proper thing to do was to make it propagate a hard error all the way up and all implementations do exactly that. In this way, behavior is standardized for all common operations. Should graph databases, beginning with Neo, undergo a similar process? Such an effort would give definite answers to most such problems, for example the forced/cascading deletion issue mentioned before. On the other hand, maybe I am overthinking this. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Cats and Dogs, living together
Would anybody be willing to share experiences with trying to introduce Neo4j into a system with another relational (or other NoSQL) database? Let me contribute my 2 cents here. I have developed a rough demo that shows what my thoughts are for integration of Neo4j instances in a managed environment. Using this JCA 1.6 compliant connector module I have successfully - Injected a ConnectionFactory (akin to a DataSource from JDBC) that returns NeoConnections (similar to javax.sql.Connection). - Performed operations on it (node creations/property settings) in tandem with operations on a mySQL instance. That is, in the same EJB method, with a container managed transaction. Both resources were enlisted in the top level tx and were successfully committed or rolledback (when I violated constraints in either db, for instance), in proper 2PC manner. So far, based on my experience working with JDBC, the feeling is the same, as far as usability goes. The GraphDatabaseService interface is not yet completely available. The code is available at the repository: The transaction manager to use (based on previous work): https://svn.neo4j.org/laboratory/users/cgioran/ContainerProvidedTxManager/ The connector module: https://svn.neo4j.org/laboratory/users/cgioran/NeoJCA/ The modified kernel (nothing major, just exposed retrieval of XAResources and enforced the new txm) https://svn.neo4j.org/laboratory/users/cgioran/neo4j-kernel-jta/ The idea is that you build/install all three above, add the jta kernel and the txm service in your ear's libraries and the connector module as an ear connector module (in application.xml). Create a pool and bind it to the JNDI and retrieve it from there in your EJB/Servlet. DISCLAIMER: The above code is a toy and completely temporary. I have not verified its operation in any significant way, I do not even know if (in fact I doubt that) the pooling mechanism works properly. Do not use it in any way that might be remotely important. The connector will work only in glassfish. Other app servers (JonAS and Geronimo for certain) require additional descriptor files apart from ra.xml If you want to deploy in another container, please consult its manual on the proper procedure. I will now go and write a post explaining in more detail how to build, install and use the above. If, in the meantime, someone feels adventurous and decides to look at the code, feedback is always welcome. The important thing for me is not the correctness of the code - there is not much there anyway. What I am interested in is its dynamics and the possible interactions in an app server. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Cats and Dogs, living together
I will now go and write a post explaining in more detail how to build, install and use the above. And it is available at http://digitalstain.blogspot.com/2010/12/towards-neo4j-connector-implementation.html be gentle. thank you for your time, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] JTA support for Neo
On Mon, Nov 22, 2010 at 4:10 PM, Johan Svensson jo...@neotechnology.com wrote: This looks great, very good work! I would like to get this merged into trunk after we releases 1.2 (not after this iteration but after the next one). The changes looks minimal to me and hopefully there are no problems going forward with the current design. Looking forward to the guide so me and others can try this out. Would it be possible to continue investigate this and see how these changes would work in a Spring environment? The goal would be to run on an external TM, in Spring (using annotated transactions) and have both Neo4j and MySQL (or some other resource) participate in the same transaction. Regards, Johan Currently I am working on a branch of the recovery robustness suite that runs the jta kernel with a jotm instance. Works like a charm, after some more successful cycles I will commit it at my laboratory space, probably within the hour. This will contain a completely setup environment for running an example and the guide comes next. If this test is considered enough, I could delay integrating my JDI-based test suite and continue work on the JCA module. Personally, I feel confident with the robustness suite, so if there are no objections until I am done with the guide, I will begin with the connector. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] JTA support for Neo
Ok, I have written a small post describing how this thing comes together. It is available at http://digitalstain.blogspot.com/2010/11/using-jotm-as-transactionmanager-in.html I hope it is clear enough. Questions asked here will end up as corrections to this. However, let me point out that here: https://svn.neo4j.org/laboratory/users/cgioran/recovery-robustness/ is a branched project that contains everything that is needed to run the JOTM enabled Neo, provided you have installed the jta kernel and the jotm service implementation. All configuration and dependencies are taken care of (the JOTM configuration is in the resources/ directory and the JVM parameters in the run-one script). Thank you for your time. CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] JTA support for Neo
IMHO you should start a branch in the SVN so others can look at the code. So, at https://svn.neo4j.org/laboratory/users/cgioran/neo4j-kernel-jta/ you can find the kernel component with my changes incorporated. The classes added are org.neo4j.kernel.impl.transaction.TransactionManagerImpl org.neo4j.kernel.impl.transaction.TransactionManagerService that are the hooks for providing custom transaction managers. The first is an extension of javax.transaction.TransactionManager adding support for startup and shutdown, an operation present in all tx managers but not part of their API. This provides the ability to plugin custom implementations in the TxModule. The second is a convenience class that is extended by tx managers that are to be provided as a service. Also, changes are present in org.neo4j.kernel.impl.transaction.TxModule for using this new way of doing things, org.neo4j.kernel.impl.transaction.TxManager org.neo4j.kernel.impl.transaction.ReadOnlyTxManager for them to fit in this and org.neo4j.kernel.EmbeddedGraphDbImpl org.neo4j.kernel.Config to bind them. This fork is (or should be) completely compatible with the official kernel, so it can be used as a drop in replacement. Any deviation is a bug and if reported it will be fixed. The second project is at https://svn.neo4j.org/laboratory/users/cgioran/JOTMService/ and is a sample implementation of a tx manager service for JOTM. To use this, build it, add the resulting jar to your classpath and, if you are using the new jta fork of the kernel, you can pass a configuration parameter of tx_manager_impl=jotm to your EmbeddedGraphDatabase and presto!, if all is well you will be using a JOTM TxManager to do your thing. Of course, the jotm libraries must be also in your classpath, version 2.1.9 If this way of doing things is met with approval, I will write a complete guide to using the above, implementation and design details and as a result a how to for adding more external tx managers. There is more to come. CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] JTA support for Neo
Chris, This sounds like some major progress! Meh. It was some work for me, mainly making the vm crash at will and getting JOTM to enlist/recover the resource managers. Things worked out of the box from the side of neo. IMHO you should start a branch in the SVN so others can look at the code. That would be the ideal way, yes. Would you set up a laboratory space for me? Having the compatibility tests is great - there are other breaking test that Mattias and Johan are using to purposely break recovery on both the normal server and HA setups. They are run on a number of amazon instances and it would be great if your tests could be integrated and run there, too, so we can find failing recovery scenarios. I assume you are referring to qa/recovery-robustness. I haven't studied its mechanics yet but I could put it there, I suppose. From a cursory look I understood that it runs from a script so my tests could fit there. any suggestions on making the use of external TX managers easier or document the setup process somewhere? For my part I could post an article on my blag(TM) on how I have setup JOTM as a Neo service and how to enable it. I would prefer to do that however after the code has been seen from eyeballs other than mine and actually certified not to be a big, fat, slobbering mess. This way the API will be standardized and people will have something concrete to read about. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] JTA support for Neo
If you can get the above test case working using other TMs together with Neo4j would be great! OK, it took some time, since I messed with some things for the first time. I have created the following: - I have added two classes in the kernel module that provide support for registration of TransactionManager implementations as services. Like the lucene indexer, if the jar is in the classpath it will be picked up and made available. The configuration passed to the GraphDatabase must define the name of the TxManager to use, defaulting of course to the native one. These changes required some modifications in TxManager, ReadOnlyTxManager, TxModule, EmbeddedGraphDbImpl and Config. - A sample implementation of the above for providing a JOTM instance as a TransactionManager. - A test platform that, through JDI crashes the system at specific points, up to a resolution of line of code. This way at any point in the transaction the system can be brought down in flames. This aspect is still a little awkward to integrate since a new JVM process must be started and terminated. At the time being I use it as a standalone test (launched at the console). Using the above, after crashing a neo instance that uses a lucene XaResource and (of course) a Neo XaResource at any point during commit or rollback, using JOTM 2.1.9, restating the system brings it in a consistent state, committing all pending resources or rolling back, whatever is needed. There are many test cases still to be implemented to make sure that the setup is production ready but what I have tested so far passes. This includes of course the stock tests for the kernel component, ensuring (to whatever possible extent) that my changes do not mess up with normal operations. - Find the least intrusive way of making neo fit in the picture, in terms of configuration/code changes etc, approve and commit those. See above. What I would like as feedback is ways the community would integrate these capabilities in their environment. This way I can provide integration methods that will make the procedure less risky and more transparent, as well as develop test cases that do not at the moment occur to me. As this feature requires changes in the neo kernel, I am still searching for the best way to make the code available. I will inform you when that changes, hopefully soon. Also, if there is interest, I could write an article on how this is implemented. What I will test next is how this thing plays with JDBC, seeing as this would be a common case. Also, I have read up on JCA and there is some preliminary code to produce a .rar archive that will provide neo functionality within a container, although this still very premature. Thank you for your time, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] JTA support for Neo
OK, I have a question. The XA/Open spec in paragraph 2.3 states that the TM can discard its knowledge of the current global tx after the RMs have committed or rolled back. Neo takes advantage of this fact in several places, where during the completion of the current tx either way (commit or rollback) it asks the TM for the current tx to retrieve related resources. In fact, the TxManager class that is provided with Neo as a top level TM forgets (i.e. unmaps the tx from the thread) *after* calling afterCompletion() hooks, so everything works fine out of the box. JOTM on the other hand is not consistent (and, eventually, spec compliant), at least this is what is seems to me. Let me explain. JOTM (its Current class, that is the TM implementation) has a commit() method that first asks RMs to commit() and then dissociates the global tx from the thread. But on rollback, first it dissociates the tx from the thread and then, from a method local copy it asks the RMs to rollback(). This leaves any RM to get a null reference when asking the JOTM TM for the current tx, which leads to failures in the process. If the rollback() code in Current is changed to first ask the RMs to rollback() and then forget the global tx, everything works as it should. My point is that, from a first view, this is a bug in JOTM, not in Neo. Does anyone have any experience with this? Am I interpreting this wrong or should it be taken upstream? cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] JTA support for Neo
Hi people, I have started work on providing support for integrating Neo in a JTA environment. This is the first time I have done something like this so some feedback would be welcome. My setup is this: I have an ApacheDS instance running to provide an LDAP service to store objects. I have tweaked (very slightly) a standalone JOTM instance to provide its TransactionManager via JNDI to the directory (I prefer that over RMI). I then start an EmbeddedGraphDatabase and replace the native neo TxManager with the JOTM provided implementation as I retrieve it from the directory, both in the PersistenceModule and the TxModule. From there, instead of asking the graphDb to beginTx() I ask the JOTM TransactionManager to begin(). I perform some Node creations and they are persisted on commit(). Also, via a debugger, I can see that the XAResources of Neo are indeed enlisted on the remove Transaction. I know that this is very premature and for the time being the changes are very rough but I think I have a proof of concept. Comments? Does anyone have any experience in a similar setting? Next steps? This has a very high coolness factor, btw :) cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] JTA support for Neo
Chris, Awesome! I think the next step would be to start testing things when neo4j needs to recover, rollback etc, I think this is where the problems arise :) Also, any chance of making a maven project out of it and having the project as a test component somewhere in the Svn repo, so it can be run as part of the QA process for releases etc? OK, so the plan I have in my mind is this: - Do runs to see what problems exist when rolling back/starting in recovery mode with the only resource being neo. - See how the whole thing works when another XA compatible resource is thrown in the mix, probably a RDBMS, checking that 2PC works. - Find the least intrusive way of making neo fit in the picture, in terms of configuration/code changes etc, approve and commit those. - Write test cases and a maven project so that it can be integrated in the release cycle to be checked for correct functionality. - After that probably I would like to fill in the gaps so that from an app server I can do a container managed tx over a jdbc connection and a neo connection. After all, this is the ultimate purpose of this exercise. I will fill you in as I go through each of the above. Thanks for your time. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] An attempt at documenting the internals and architecture of Neo
Hi people, the last part of my series: Neo4j Internals: Transactions (Part 3) as a complete run and a conclusion is available here: http://digitalstain.blogspot.com/2010/11/neo4j-internals-transactions-part-3-as.html I try, by following a very small example, to demonstrate the interactions that lead to a transactional operation and at the same time show how the various parts I have described previously work together to store a node. I think I am done writing on Neo internals for some time, I now know enough to start meddling with the code. I hope you found my work useful and that you enjoyed it as much as I did writing it. If there are any questions or subjects that you feel are underexposed or wrong, or if you would like to discuss something, this mailing list or the #neo4j channel on freenode is where I encourage you to do it, so that others can benefit as well. Thank you for your time. CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] An attempt at documenting the internals and architecture of Neo
The second-to-last post in the core series: Neo4j Internals: Interlude - Xa roundup and consistency available here: http://digitalstain.blogspot.com/2010/11/neo4j-internals-interlude-xa-roundup.html A needed coverage of some classes that are a core part of the Neo mechanism. It should have been posted before I started the tx discussion but I had not understood their details yet. Anyway, this is kind of boring text, dealing mostly with engineering details and hardly with any algorithms, discussing mainly the Xa related classes. The second part is the LockReleaser and how it achieves a consistent view of Primitives within a tx. I hope you will enjoy it and find it helpful. The next post is under construction and delivers what this series was supposed to: A walkthrough from db boot-up to shutdown. As always, all comments welcome. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Delete a RelationshipType
I am curious, are there any 'developer tricks' for influencing when old slots are reused? For example, instead of adding the new relationships immediately after deleting old ones, but keeping a cache of 'relationships to add' and adding them a little later. Will that increase the chance of old slots being reused? My understanding is this: An id for every record and, correspondingly, a number of bytes in the file, are reserved the moment that, during a tx, a new entity is created. Conversely, the id of an entity is freed upon successful commit() of the tx that deleted it, although the space is still occupied but marked as free and available to be returned at the next request for a new entity. Note that all these operations have entity type scope, meaning that a freed Node does not in any way influence Relationship operations. The above lead to a strategy of performing and committing any deletes first and then, in a new tx, create all that is needed. If, during the same tx, a delete is made and then a create, the deleted id is not reused. I think this is relevant to cases where the developer wants to keep the database reasonably compact but does large amounts of deletion and creation. I think there was a similar discussion here some time ago and I seem to recall that the proposed solution for compacting a db after a long streak of deletes/inserts was to recreate the db in a new location. Hope that helps somewhat. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Delete a RelationshipType
My understanding is this: And it is wrong. Consider the following snippet: GraphDatabaseService graphDb = new EmbeddedGraphDatabase(var/base); Transaction tx = graphDb.beginTx(); Node firstNode = graphDb.createNode(); Node secondNode = graphDb.createNode(); System.out.println(firstNode.getId()); System.out.println(secondNode.getId()); secondNode.delete(); Node thirdNode = graphDb.createNode(); System.out.println(thirdNode.getId()); tx.success(); tx.finish(); tx = graphDb.beginTx(); Node fourthNode = graphDb.createNode(); System.out.println(fourthNode.getId()); tx.success(); tx.finish(); graphDb.shutdown(); If the id pool was reused after a successful commit() (success() in the above code) as I described then, on a new db, this should print 1 2 3 2 but instead it prints 1 2 3 4 The id 2 is reused upon restart of the db. That will teach me to speak without checking first. What I did wrong is left as an exercise for tomorrow. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Indirection
On Sat, Oct 30, 2010 at 12:51 PM, Andres Taylor andres.tay...@neotechnology.com wrote: Guys, I'm trying to grokk the code. In a couple of places, there are indirections that I don't understand. Like EmbeddedGraphDatabase, which is, as far as I can see, mostly a thin wrapper around EmbeddedGraphDatabaseImpl. Why is this? Andrés I second that. It isn't like there is a factory in EmbeddedGraphDatabase that will provide different implementations of something based on the Config (besides, the read-only vs the full r/w db issue is handled differently). The EmbeddedGraphDatabaseImpl field is hardwired there. I can only assume this was supposed to (or will?) augment the GraphDatabaseService interface with operations other that the EmbeddedGraphDatabaseImpl ones, given that the latter does not implement that. I also have difficulty grasping the EventConsumers concept. They are created in the XaConnection only to be stored in the NioNeoDbPersistenceSource.NioNeoDbResourceConnection and then used to forward calls about primitives. Why is that level of abstraction necessary? I think it would be cleaner and without loss of extensibility for the ResourceConnection to forward directly to the XaConnection. But I admit there are many things happening there, I may be missing something. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] An attempt at documenting the internals and architecture of Neo
Another post in the same series Neo4j Internals: Transactions (Part 2) - XaResources, Transactions and TransactionManagers available at http://digitalstain.blogspot.com/2010/10/neo4j-internals-transactions-part-2.html As the posts go by, I tend to have an intuitive feel about the code, so I may presume some things as obvious while they are not. Please, inform me of such shortcomings along with any errors you find. Also, if you have any comments on the wording, the format or the content, it would be a great help. Thank you for your time. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] An attempt at documenting the internals and architecture of Neo
Hi people, here is the next part of my series on Neo internals Neo4j Internals: Transactions (Part 1) - Write Ahead Log and Deadlock Detection available at http://digitalstain.blogspot.com/2010/10/neo4j-internals-transactions-part-1.html Since the tx handling code is much more complex than the previously described components, I have omitted many details, trying to show only how the basic things are done. Even so, it was proved that there are many things to write about, so I had to break down the tx description to many parts. In this first one, the write ahead log and the deadlock detection mechanisms are described, somewhat coarsely. As always, any mistakes you find, please take some time to point them out so I can have them fixed. Thank you for your time and your positive comments. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] An attempt at documenting the internals and architecture of Neo
On Mon, Oct 18, 2010 at 11:57 PM, Peter Neubauer peter.neuba...@neotechnology.com wrote: Chris, great reading your blogs on this! Did you get on with the transaction piece yet? In case you need info, don't hesitate to ask here on the list, there are a number of folks that have disected the area before :) Also, when it comes to Neo4j working with other TX managers in a JTA environment like JOTM and Atomikos, I hope we will be able at least to start with failing tests in these environments in the next iteration and beginning with that, start working around the issues that JTA is leaving for non-relational stores like Neo4j. There might be some details that you are interested in when digging there :) Cheers, /peter neubauer Peter, thanks for your input. The transaction handling code is the aspect of a db that I am least familiar with from an implementation standpoint, so it took a bit more time than I had planned. I have come through, however, wiser and ready to write about it in what will in all probability be more than one post. Soon :) As for JTA support, it is one of the reasons I started my documentation efforts and I would like to see neo working in a JTA/JTS environment. But first I have to write about XALogicalLog and its friends. Stay tuned. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] EmbeddedGraphDatabase shutdown leaves a Timer thread running
On Wed, Oct 13, 2010 at 4:11 PM, Adam Lehenbauer a...@fixflyer.com wrote: I think I have the Getting Started example working, but after it creates the nodes and shuts down, my JVM won't exit because there is a non-daemon Timer thread running. I have an example class with a main() that is copied near-verbatim from http://wiki.neo4j.org/content/Getting_Started_Guide except for the db path and a few extra System.outs. Everything seems to work normally, but after shutting down the EmbeddedGraphDatabase the JVM will not exit. A jstack shows a running java.util.TimerThread, which I assume is started by the database. Its stack is: --- Timer-0 prio=10 tid=0x7f59c0056800 nid=0xa29 in Object.wait() [0x7f59bf6d5000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x7f5a1ed37270 (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:509) - locked 0x7f5a1ed37270 (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) --- I'm using the neo4j maven dependency: org.neo4j.neo4j, version=1.2.M01, type=pom. When I changed this to neo4j-kernel, version=1.0 the JVM exited as expected (no code change on my end). Does anyone have any idea why this is happening? Adam Hi there, Could you run it with a configuration setting of cache_type=soft and inform on the result? Also, describe a bit your environment (CPU, OS, JVM) cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] [SPAM] Re: EmbeddedGraphDatabase shutdown leaves a Timer thread running
On Wed, Oct 13, 2010 at 5:34 PM, Rick Bullotta rick.bullo...@burningskysoftware.com wrote: Probably the recently added phone home functionality (UDC). Just a guess, though. in that case removing the dependency as per http://wiki.neo4j.org/content/UDC should make it work. My setup is the same bar a _21 JVM and 2 cores. All my tests terminate normally. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] An attempt at documenting the internals and architecture of Neo
Hi list, I have another couple of posts on the same series. The first can be read at http://digitalstain.blogspot.com/2010/10/neo4j-internals-file-storage.html and it is an exposition of my understanding of the format of the files that neo keeps its data in. It is a fairly big post that deals with small details, so should anyone read it do not be surprised if it proves to be reality incompatible. Corrections will be incorporated ASAP. The second is about the next level in the persistence subsystem of neo and is here: http://digitalstain.blogspot.com/2010/10/neo4j-internals-persistence-and-memory.html I try to explain how the buckets of bits are translated to entities in the domain of neo, how the memory mapping works and I dwell for a bit over the light vs heavy issue. Note that both posts where written in one stretch each after I finished going through the code. I expect that at some points I may have left out details that are not so clear for someone going through the implementation for the first time. If you note something like that (or any other errors, of course) please comment so that I can remedy them. Hopefully within the week I will have a piece about the transaction handling and how that interfaces with the store and the NodeManager. But to get there I have some reading to do. cheers, CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] An attempt at documenting the internals and architecture of Neo
On Thu, Oct 7, 2010 at 2:59 PM, Andres Taylor andres.tay...@neotechnology.com wrote: Hi Chris, Absolutely awesome stuff you have written. As a new hire in Neo Technologies, I'm reading your blog posts with great interest. I like how you take code and make a story out of it - it's much easier to understand the code after reading your posts on it. Thank you! I am happy to know that it proved useful to someone. Thank you. I have two wishes that would make your posts even more awesomer (for me at least). 1. I'd love it if you could point to the code. In this post, you write about getNodeById() - why not make the method name a link to the class file in https://svn.neo4org? Fair enough. I had decided against it because I don't like to litter the text with links and, given the expectation that there would be many references, I feared it would be distracting. Moreover, if someone is reading my posts I assumed she has already downloaded the code and set it up in her IDE. However, I think I should add links to at least some of the referenced code, to save some trouble and link back to the project website. 2. I think that it would be interesting to have a post about how backups are done. Personally, I have always been interested in how you make a backup of a live database without using huge locks. But maybe that's just me... :) Nice idea. Added in the TODO list. You are doing a great job Chris. If you have any questions, ask away here on the list. Appreciate it. Andrés CG ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
