[Neo4j] Can not recover from a HA failure as a BlockingReadTimeoutException
Dear List, I am using 1.6.1 and embedded HA mode. Do u have any ideas that how to resolve below error? =2012-05-15 16:07:10,453 main ERROR - [ServletCntxListener.moreStart] Failed in start the graph java.lang.ExceptionInInitializerError at com.hlt.neo.CountryFactory.init(CountryFactory.java:60) at com.hlt.neo.CountryFactory.clinit(CountryFactory.java:49) at com.hlt.online.ServletCntxListener.moreStart(ServletCntxListener.java:44) at com.gr.online.servlet.InitServletCntxListener.contextInitialized(InitServletCntxListener.java:44) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4205) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4704) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) at org.apache.catalina.core.StandardHost.start(StandardHost.java:840) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) at org.apache.catalina.core.StandardService.start(StandardService.java:525) at org.apache.catalina.core.StandardServer.start(StandardServer.java:754) at org.apache.catalina.startup.Catalina.start(Catalina.java:595) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: java.lang.RuntimeException: Tried to join the cluster, but was unable to at org.neo4j.kernel.HAGraphDb.startUp(HAGraphDb.java:297) at org.neo4j.kernel.HAGraphDb.init(HAGraphDb.java:167) at org.neo4j.kernel.HAGraphDb.init(HAGraphDb.java:131) at org.neo4j.kernel.HighlyAvailableGraphDatabase.init(HighlyAvailableGraphDatabase.java:55) at com.gr.neo.Graph.init(Graph.java:64) at com.gr.neo.Graph.init(Graph.java:45) at com.gr.neo.Graph.clinit(Graph.java:42) ... 19 more Caused by: java.lang.RuntimeException: Gave up trying to copy store from master at org.neo4j.kernel.HAGraphDb.getFreshDatabaseFromMaster(HAGraphDb.java:237) at org.neo4j.kernel.HAGraphDb.startUp(HAGraphDb.java:273) ... 25 more Caused by: org.neo4j.com.ComException: org.jboss.netty.handler.queue.BlockingReadTimeoutException at org.neo4j.com.DechunkingChannelBuffer.readNext(DechunkingChannelBuffer.java:75) at org.neo4j.com.DechunkingChannelBuffer.readNextChunk(DechunkingChannelBuffer.java:93) at org.neo4j.com.DechunkingChannelBuffer.init(DechunkingChannelBuffer.java:59) at org.neo4j.com.Client.sendRequest(Client.java:210) at org.neo4j.com.Client.sendRequest(Client.java:185) at org.neo4j.kernel.ha.MasterClient.copyStore(MasterClient.java:268) at org.neo4j.kernel.HAGraphDb.copyStoreFromMaster(HAGraphDb.java:425) at org.neo4j.kernel.HAGraphDb.getFreshDatabaseFromMaster(HAGraphDb.java:224) ... 26 more Caused by: org.jboss.netty.handler.queue.BlockingReadTimeoutException at org.jboss.netty.handler.queue.BlockingReadHandler.readEvent(BlockingReadHandler.java:236) at org.jboss.netty.handler.queue.BlockingReadHandler.read(BlockingReadHandler.java:167) at org.neo4j.com.DechunkingChannelBuffer.readNext(DechunkingChannelBuffer.java:66) ... 33 more ___ NOTICE: THIS MAILING LIST IS BEING SWITCHED TO GOOGLE GROUPS, please register and consider posting at https://groups.google.com/forum/#!forum/neo4j Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] HA - Unable to read additional data from server sessionid 0x0
Thanks a lot, Peter. I found another way to bring it up again, As I thought there should be some incorrect status stored in the graph, maybe start up the master instance in Non-HA mode can rectify the status. I tried and that works. Now the HA cluster is running normally. However, to prevent it happen again, I will setup a 3-node zk cluster. Thanks a lot~ Thanks and Regards, Samuel 2012/3/14 Peter Neubauer peter.neuba...@neotechnology.com Hi there, I talked to Chris Gioran and got the following diagnostics: This* is what happens when in a quorum of 2 ZK instances one goes down. (*this == the remaining ZK instance - which is not in standalone mode - will refuse to accept requests, hence the ZooClient in neo will barf and die). Reproduced it in exactly the same manner: started a 2 ZK quorum and 2 neo instances using it, brought down one of the zk servers and restarted one neo instance - it refused to come online again. If that is what you did, then that is what happened. To bring it up again, I think you need to get a 3 - node ZK cluster up and running again before it will accept new cluster nodes again. Cheers, /peter neubauer G: neubauer.peter S: peter.neubauer P: +46 704 106975 L: http://www.linkedin.com/in/neubauer T: @peterneubauer Neo4j 1.6 released - dzone.com/6S4K The Neo4j Heroku Challenge - http://neo4j-challenge.herokuapp.com/ On Mon, Mar 12, 2012 at 2:00 PM, Samuel Feng okos...@gmail.com wrote: Dear List, I am using neo4j 1.6.1 and embedded HA Mode. The cluster(with 2 servers) works normally at the beginning. However, after one server crashed, the cluster can not start up and keep return below logs. Seems both of the two servers are trying to connect each other but failed again again. I tried to remove the version-2 folder to reset the cluster. But the problem exits. Could you please help me how to recover the cluster? Thanks a lot =2012-03-12 20:58:45,946 main-SendThread(125.65.108.89:2181) INFO - [ClientCnxn$SendThread.startConnect] Opening socket connection to server 125.65.108.23/125.65.108.23:2181 =2012-03-12 20:58:45,947 main-SendThread(125.65.108.23:2181) INFO - [ClientCnxn$SendThread.primeConnection] Socket connection established to 125.65.108.23/125.65.108.23:2181, initiating session =2012-03-12 20:58:45,950 main-SendThread(125.65.108.23:2181) INFO - [ClientCnxn$SendThread.run] Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect =2012-03-12 20:58:46,066 main-SendThread(125.65.108.89:2181) INFO - [ClientCnxn$SendThread.startConnect] Opening socket connection to server 125.65.108.23/125.65.108.23:2181 =2012-03-12 20:58:46,067 main-SendThread(125.65.108.23:2181) INFO - [ClientCnxn$SendThread.primeConnection] Socket connection established to 125.65.108.23/125.65.108.23:2181, initiating session =2012-03-12 20:58:46,068 main-SendThread(125.65.108.23:2181) INFO - [ClientCnxn$SendThread.run] Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect =2012-03-12 20:58:46,153 main-SendThread(125.65.108.23:2181) INFO - [ClientCnxn$SendThread.startConnect] Opening socket connection to server 125.65.108.89/125.65.108.89:2181 =2012-03-12 20:58:46,157 main-SendThread(125.65.108.89:2181) INFO - [ClientCnxn$SendThread.primeConnection] Socket connection established to 125.65.108.89/125.65.108.89:2181, initiating session =2012-03-12 20:58:46,160 main-SendThread(125.65.108.89:2181) INFO - [ClientCnxn$SendThread.run] Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect =2012-03-12 20:58:46,802 main-SendThread(125.65.108.23:2181) INFO - [ClientCnxn$SendThread.startConnect] Opening socket connection to server 125.65.108.89/125.65.108.89:2181 =2012-03-12 20:58:46,806 main-SendThread(125.65.108.89:2181) INFO - [ClientCnxn$SendThread.primeConnection] Socket connection established to 125.65.108.89/125.65.108.89:2181, initiating session =2012-03-12 20:58:46,812 main-SendThread(125.65.108.89:2181) INFO - [ClientCnxn$SendThread.run] Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect Thanks a lot, Samuel ___ NOTICE: THIS MAILING LIST IS BEING SWITCHED TO GOOGLE GROUPS, please register and consider posting at https://groups.google.com/forum/#!forum/neo4j Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ NOTICE: THIS MAILING LIST IS BEING SWITCHED TO GOOGLE GROUPS, please register and consider posting at https://groups.google.com/forum
[Neo4j] HA - Unable to read additional data from server sessionid 0x0
Dear List, I am using neo4j 1.6.1 and embedded HA Mode. The cluster(with 2 servers) works normally at the beginning. However, after one server crashed, the cluster can not start up and keep return below logs. Seems both of the two servers are trying to connect each other but failed again again. I tried to remove the version-2 folder to reset the cluster. But the problem exits. Could you please help me how to recover the cluster? Thanks a lot =2012-03-12 20:58:45,946 main-SendThread(125.65.108.89:2181) INFO - [ClientCnxn$SendThread.startConnect] Opening socket connection to server 125.65.108.23/125.65.108.23:2181 =2012-03-12 20:58:45,947 main-SendThread(125.65.108.23:2181) INFO - [ClientCnxn$SendThread.primeConnection] Socket connection established to 125.65.108.23/125.65.108.23:2181, initiating session =2012-03-12 20:58:45,950 main-SendThread(125.65.108.23:2181) INFO - [ClientCnxn$SendThread.run] Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect =2012-03-12 20:58:46,066 main-SendThread(125.65.108.89:2181) INFO - [ClientCnxn$SendThread.startConnect] Opening socket connection to server 125.65.108.23/125.65.108.23:2181 =2012-03-12 20:58:46,067 main-SendThread(125.65.108.23:2181) INFO - [ClientCnxn$SendThread.primeConnection] Socket connection established to 125.65.108.23/125.65.108.23:2181, initiating session =2012-03-12 20:58:46,068 main-SendThread(125.65.108.23:2181) INFO - [ClientCnxn$SendThread.run] Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect =2012-03-12 20:58:46,153 main-SendThread(125.65.108.23:2181) INFO - [ClientCnxn$SendThread.startConnect] Opening socket connection to server 125.65.108.89/125.65.108.89:2181 =2012-03-12 20:58:46,157 main-SendThread(125.65.108.89:2181) INFO - [ClientCnxn$SendThread.primeConnection] Socket connection established to 125.65.108.89/125.65.108.89:2181, initiating session =2012-03-12 20:58:46,160 main-SendThread(125.65.108.89:2181) INFO - [ClientCnxn$SendThread.run] Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect =2012-03-12 20:58:46,802 main-SendThread(125.65.108.23:2181) INFO - [ClientCnxn$SendThread.startConnect] Opening socket connection to server 125.65.108.89/125.65.108.89:2181 =2012-03-12 20:58:46,806 main-SendThread(125.65.108.89:2181) INFO - [ClientCnxn$SendThread.primeConnection] Socket connection established to 125.65.108.89/125.65.108.89:2181, initiating session =2012-03-12 20:58:46,812 main-SendThread(125.65.108.89:2181) INFO - [ClientCnxn$SendThread.run] Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect Thanks a lot, Samuel ___ NOTICE: THIS MAILING LIST IS BEING SWITCHED TO GOOGLE GROUPS, please register and consider posting at https://groups.google.com/forum/#!forum/neo4j Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] [Neo4j-User] Queries about HA
Thanks Peter, I studied how the HA works in http://docs.neo4j.org/chunked/snapshot/ha-how.html. It mentioned that If the new master is elected and performs changes before the old master recovers, there will two different versions of the data. The old master will not be able to attach itself to the cluster and will require maintenance (replace the wrong version of the data with the one running in the cluster. Say, I have two neo4j instances(A B) running, When instance A crashed, B continues to work and has many writes on it. Then Server A is recovered but it can not attach itself to the cluster as the wrong data version. So I need to manually replace the instance A's data using B's. But the graph size is around 4G, I have to copy all the 4G data to server A? Or any other simple and fast ways? Regards, Samuel 2011/12/6 Peter Neubauer peter.neuba...@neotechnology.com Samuel, I would recommend HA for your usecase, since then the Neo4j cluster will handling new neo4j instances appearing and copying data etc. Look at http://docs.neo4j.org/chunked/snapshot/ha.html for setup docs. There seems to be some interest in RMI remoting to neo4j. However, we think this is not the ultimate way to go in the long run since it is Java-centric and still not fast and stable enough for want we want to achieve. However, if the community finds it useful, I am all up for someone stepping up and invest the last mile to make it good, maybe put it on GIThub for easier collaboration, of course. Tobias original code is at https://svn.neo4j.org/components/remote-graphdb/trunk/, and the API should not have moved terribly out of sync, so I guess it is not a LOT of work to at least get it working. Anyone? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer brew install neo4j neo4j start heroku addons:add neo4j On Tue, Dec 6, 2011 at 4:43 AM, Samuel Feng okos...@gmail.com wrote: Dear List, I'm using neo4j 1.5GA and using embeddedGraphDatabase to access the graph. Now the graph size is around 4G. Now I would like to try HA mode and have several questions. 1) Can I access the graph in some kind of remote mode? So that I can use many tomcat servers to access the graph remotely and do not need to change any code. I understand neo4j can run as a server, however, we can only use REST api to accessing the neo4j server, this will cause many many code changes for me. Please correct me if I have any misunderstanding. Seems there is a neo4j-remote-graphdb component which uses RMI to access a graph remotely. BUT in the wiki http://components.neo4j.org/neo4j-remote-graphdb/stable/, you mentioned The implementation is not speedy enough to be used as the main access channel to a Graph Database for actual applications. 2) If I use neo4j embedded in HA mode, say, I have two neo4j instances(A B) running, When instance A crashed, B continues to work and has many writes on it. Then Server A is recovered but it can not attach itself to the cluster as the wrong data version. So I need to manually replace the instance A's data using B's. But the graph size is around 4G, I have to copy all the 4G data to server A? Or any other simple and fast ways? Thanks and Regards, Samuel ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- You received this message because you are subscribed to the Google Groups Neo4j User Discussions group. To post to this group, send email to u...@neo4j.org. To unsubscribe from this group, send email to user+unsubscr...@neo4j.org . For more options, visit this group at http://groups.google.com/a/neo4j.org/group/user/?hl=en. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Queries about HA
Dear List, I'm using neo4j 1.5GA and using embeddedGraphDatabase to access the graph. Now the graph size is around 4G. Now I would like to try HA mode and have several questions. 1) Can I access the graph in some kind of remote mode? So that I can use many tomcat servers to access the graph remotely and do not need to change any code. I understand neo4j can run as a server, however, we can only use REST api to accessing the neo4j server, this will cause many many code changes for me. Please correct me if I have any misunderstanding. Seems there is a neo4j-remote-graphdb component which uses RMI to access a graph remotely. BUT in the wiki http://components.neo4j.org/neo4j-remote-graphdb/stable/, you mentioned The implementation is not speedy enough to be used as the main access channel to a Graph Database for actual applications. 2) If I use neo4j embedded in HA mode, say, I have two neo4j instances(A B) running, When instance A crashed, B continues to work and has many writes on it. Then Server A is recovered but it can not attach itself to the cluster as the wrong data version. So I need to manually replace the instance A's data using B's. But the graph size is around 4G, I have to copy all the 4G data to server A? Or any other simple and fast ways? Thanks and Regards, Samuel ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] About Neo4j Indexing
Hello List,
Anyone can help me on this?
Thanks and regards,
Samuel
在 2011年11月14日 下午1:51，Samuel Feng okos...@gmail.com写道：
Dear List,
I have two questions about indexing
*Question 1*
At the time of creation, we can use extra configuration can be specified
to control the behavior of the index and which backend to use.
e.g,
IndexManager index = graphDb.index();
IndexNode movies = index.forNodes( movies-fulltext,
MapUtil.stringMap( IndexManager.PROVIDER, lucene, analyzer,
org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer ) );
movies.add( theMatrix, cTitle, 黑客帝国 );
movies.add( theMatrix, date, 2000-01-01 );
When adding node theMatrix to index, all the values will be
analyzed/tokenized by SmartChineseAnalyzer.
However， for some fields I do not want it to be analyzed/tokenized, Any
interfaces to implement this? Can enhance valueContext so that I can pass
in something like Field.Index.NOT_ANALYZED when adding a node into index?
*Question 2*
For Query,
IndexHitsNode nodes = movies .query(new BooleanQuery(...));
Node currentNode = null;
ListMovie result = new ArrayListMovie();
while (nodes.hasNext()) {
currentNode = nodes.next();
Movie m = new Movie(currentNode);
if(m.getDate().equals(2001-01-01)){
result.add(m);
}
}
I found that if the indexHits is large, say size() 2, Each
m.getDate() will spend some time to load the value from underlying
node(especially the first-time query), So the total elapsed time is very
long.
Any interface that I can read the lucene document behind this node
directly? Maybe u can use nodes.currentDoc() to expose it?
Thanks and Regards,
Samuel
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] About Neo4j Indexing
Dear List,
I have two questions about indexing
*Question 1*
At the time of creation, we can use extra configuration can be specified to
control the behavior of the index and which backend to use.
e.g,
IndexManager index = graphDb.index();
IndexNode movies = index.forNodes( movies-fulltext,
MapUtil.stringMap( IndexManager.PROVIDER, lucene, analyzer,
org.apache.lucene.analysis.cn.smart.SmartChineseAnalyzer ) );
movies.add( theMatrix, cTitle, 黑客帝国 );
movies.add( theMatrix, date, 2000-01-01 );
When adding node theMatrix to index, all the values will be
analyzed/tokenized by SmartChineseAnalyzer.
However， for some fields I do not want it to be analyzed/tokenized, Any
interfaces to implement this? Can enhance valueContext so that I can pass
in something like Field.Index.NOT_ANALYZED when adding a node into index?
*Question 2*
For Query,
IndexHitsNode nodes = movies .query(new BooleanQuery(...));
Node currentNode = null;
ListMovie result = new ArrayListMovie();
while (nodes.hasNext()) {
currentNode = nodes.next();
Movie m = new Movie(currentNode);
if(m.getDate().equals(2001-01-01)){
result.add(m);
}
}
I found that if the indexHits is large, say size() 2, Each
m.getDate() will spend some time to load the value from underlying
node(especially the first-time query), So the total elapsed time is very
long.
Any interface that I can read the lucene document behind this node
directly? Maybe u can use nodes.currentDoc() to expose it?
Thanks and Regards,
Samuel
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] InvalidRecordException
Thanks David, I am using 1.4M04. 2011/11/1 David Montag david.mon...@neotechnology.com Hi Samuel, What Neo4j version are you running? David 2011/10/31 Samuel Feng okos...@gmail.com Hello List, I encountered below exception，There should be some concurrent issue. 2011-11-01 02:01:55,882 RobotScheduler_Worker-7 ERROR - [ProductInstanceFactory.getPICount] Unexpected Exception org.neo4j.kernel.impl.nioneo.store.InvalidRecordException: Node[817] is neither firstNode[323469] nor secondNode[371464] for Relationship[992056] at org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:159) at org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:97) at org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:108) at org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:603) at org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:399) at org.neo4j.kernel.impl.core.IntArrayIterator.hasNext(IntArrayIterator.java:93) at com.heartwater.neo.ProductInstanceFactory.getPICount(ProductInstanceFactory.java:523) at com.heartwater.online.ClearingRobot.run(ClearingRobot.java:220) at com.gr.online.robot.AbstractRobot.execute(AbstractRobot.java:57) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525) Can u tell me how to correct this error？Remove the Node[817] directly? Thanks a lot. Samuel ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- David Montag david.mon...@neotechnology.com Neo Technology, www.neotechnology.com Cell: 650.556.4411 Skype: ddmontag ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] doInternalRecovery takes a long time
Thanks Tobias, I will try to upgrade my workspace to jdk1.6. http://wiki.neo4j.org/content/Java_Setup_HowTo Just a remind, I think u should arrange to change the jdk requirments in this page. 2011/3/10 Tobias Ivarsson tobias.ivars...@neotechnology.com J2SE 5.0 reached its End of Service Life (EOSL) on November 3, 2009, which is the date of the final publicly available update of version 5.0 (J2SE 5.0 Update 22). See: http://www.oracle.com/technetwork/java/javase/downloads/index-jdk5-jsp-142662.html We have decided to not support a JDK that is no longer supported by the JDK vendor. Compiling our code with the class file version set to 1.6 is our way to *strongly* urge you to move to JDK 1.6, which in every way is a better JDK. If you for some reason still need to use JDK 1.5, you could build your own, unsupported, Neo4j version. It should be possible to compile Neo4j using JDK 1.6 but setting the target version to 1.5. Most of the strict 1.6 dependencies are compile-time only. There might be a few runtime dependencies on 1.5, but those are probably limited to java.util.Deque and java.util.concurrent.TimeUnit. Your other option is to get a payed JDK 1.5 support contract for Neo4j from Neo Technology. Cheers, Tobias On Thu, Mar 10, 2011 at 8:54 AM, Samuel Feng okos...@gmail.com wrote: David, I am using jdk1.5. After upgraded to 1.3M03, I Encountered* * *java.lang.UnsupportedClassVersionError: Bad version number in .class file* Error. Seems the jars in neo4j's lib folder are complied using jdk6. Can I use 1.5 to compiled the sources directly? 2011/3/10 Samuel Feng okos...@gmail.com Thanks David. My concern is if the neo4j data in 1.2 can be used in 1.3 directly? I have used custom analyzer in my index. 2011/3/10 David Montag david.mon...@neotechnology.com Samuel, Can you try it with 1.3.M03, if possible? Thanks, David On Wed, Mar 9, 2011 at 6:13 PM, Samuel Feng okos...@gmail.com wrote: Dear list, I am developing a tomcat application in eclipse which has about 100,000 nodes using EmbeddedGraphDatabase (Neo4j version 1.2M05) Sometimes I took me a long time(more than half an hour) to doInternalRecovery, maybe because I shutdown the tomcat server directly. From the messages.log, I can find many many Injected two phase commit, *Can u tell me what cause this and how to prevent it?* Thu Mar 10 09:47:47 CST 2011: Opened [C:\home\heartwater\graph\\nioneo_logical.log.1] clean empty log, version=1 Thu Mar 10 09:47:47 CST 2011: Opened [C:\home\heartwater\graph\/lucene/lucene.log.1] clean empty log, version=0 Thu Mar 10 09:47:47 CST 2011: Opened [C:\home\heartwater\graph\/lucene-fulltext/lucene.log.1] clean empty log, version=0 Thu Mar 10 09:47:47 CST 2011: Non clean shutdown detected on log [C:\home\heartwater\graph\index/lucene.log.1]. Recovery started ... Thu Mar 10 09:47:47 CST 2011: [C:\home\heartwater\graph\index/lucene.log.1] logVersion=0 with committed tx=1 Thu Mar 10 09:47:48 CST 2011: Injected two phase commit, txId=2 Thu Mar 10 09:47:48 CST 2011: Injected two phase commit, txId=3 Thu Mar 10 09:47:48 CST 2011: Injected two phase commit, txId=4 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=5 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=6 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=7 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=8 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=9 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=10 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=11 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=12 Thu Mar 10 09:47:50 CST 2011: Injected two phase commit, txId=13 Thu Mar 10 09:47:50 CST 2011: Injected two phase commit, txId=14 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- David Montag Neo Technology, www.neotechnology.com Cell: 650.556.4411 david.mon...@neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Tobias Ivarsson tobias.ivars...@neotechnology.com Hacker, Neo Technology www.neotechnology.com Cellphone: +46 706 534857 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] doInternalRecovery takes a long time
Dear list, I am developing a tomcat application in eclipse which has about 100,000 nodes using EmbeddedGraphDatabase (Neo4j version 1.2M05) Sometimes I took me a long time(more than half an hour) to doInternalRecovery, maybe because I shutdown the tomcat server directly. From the messages.log, I can find many many Injected two phase commit, *Can u tell me what cause this and how to prevent it?* Thu Mar 10 09:47:47 CST 2011: Opened [C:\home\heartwater\graph\\nioneo_logical.log.1] clean empty log, version=1 Thu Mar 10 09:47:47 CST 2011: Opened [C:\home\heartwater\graph\/lucene/lucene.log.1] clean empty log, version=0 Thu Mar 10 09:47:47 CST 2011: Opened [C:\home\heartwater\graph\/lucene-fulltext/lucene.log.1] clean empty log, version=0 Thu Mar 10 09:47:47 CST 2011: Non clean shutdown detected on log [C:\home\heartwater\graph\index/lucene.log.1]. Recovery started ... Thu Mar 10 09:47:47 CST 2011: [C:\home\heartwater\graph\index/lucene.log.1] logVersion=0 with committed tx=1 Thu Mar 10 09:47:48 CST 2011: Injected two phase commit, txId=2 Thu Mar 10 09:47:48 CST 2011: Injected two phase commit, txId=3 Thu Mar 10 09:47:48 CST 2011: Injected two phase commit, txId=4 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=5 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=6 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=7 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=8 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=9 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=10 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=11 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=12 Thu Mar 10 09:47:50 CST 2011: Injected two phase commit, txId=13 Thu Mar 10 09:47:50 CST 2011: Injected two phase commit, txId=14 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] doInternalRecovery takes a long time
Thanks David. My concern is if the neo4j data in 1.2 can be used in 1.3 directly? I have used custom analyzer in my index. 2011/3/10 David Montag david.mon...@neotechnology.com Samuel, Can you try it with 1.3.M03, if possible? Thanks, David On Wed, Mar 9, 2011 at 6:13 PM, Samuel Feng okos...@gmail.com wrote: Dear list, I am developing a tomcat application in eclipse which has about 100,000 nodes using EmbeddedGraphDatabase (Neo4j version 1.2M05) Sometimes I took me a long time(more than half an hour) to doInternalRecovery, maybe because I shutdown the tomcat server directly. From the messages.log, I can find many many Injected two phase commit, *Can u tell me what cause this and how to prevent it?* Thu Mar 10 09:47:47 CST 2011: Opened [C:\home\heartwater\graph\\nioneo_logical.log.1] clean empty log, version=1 Thu Mar 10 09:47:47 CST 2011: Opened [C:\home\heartwater\graph\/lucene/lucene.log.1] clean empty log, version=0 Thu Mar 10 09:47:47 CST 2011: Opened [C:\home\heartwater\graph\/lucene-fulltext/lucene.log.1] clean empty log, version=0 Thu Mar 10 09:47:47 CST 2011: Non clean shutdown detected on log [C:\home\heartwater\graph\index/lucene.log.1]. Recovery started ... Thu Mar 10 09:47:47 CST 2011: [C:\home\heartwater\graph\index/lucene.log.1] logVersion=0 with committed tx=1 Thu Mar 10 09:47:48 CST 2011: Injected two phase commit, txId=2 Thu Mar 10 09:47:48 CST 2011: Injected two phase commit, txId=3 Thu Mar 10 09:47:48 CST 2011: Injected two phase commit, txId=4 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=5 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=6 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=7 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=8 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=9 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=10 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=11 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=12 Thu Mar 10 09:47:50 CST 2011: Injected two phase commit, txId=13 Thu Mar 10 09:47:50 CST 2011: Injected two phase commit, txId=14 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- David Montag Neo Technology, www.neotechnology.com Cell: 650.556.4411 david.mon...@neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] doInternalRecovery takes a long time
David, I am using jdk1.5. After upgraded to 1.3M03, I Encountered* * *java.lang.UnsupportedClassVersionError: Bad version number in .class file* Error. Seems the jars in neo4j's lib folder are complied using jdk6. Can I use 1.5 to compiled the sources directly? 2011/3/10 Samuel Feng okos...@gmail.com Thanks David. My concern is if the neo4j data in 1.2 can be used in 1.3 directly? I have used custom analyzer in my index. 2011/3/10 David Montag david.mon...@neotechnology.com Samuel, Can you try it with 1.3.M03, if possible? Thanks, David On Wed, Mar 9, 2011 at 6:13 PM, Samuel Feng okos...@gmail.com wrote: Dear list, I am developing a tomcat application in eclipse which has about 100,000 nodes using EmbeddedGraphDatabase (Neo4j version 1.2M05) Sometimes I took me a long time(more than half an hour) to doInternalRecovery, maybe because I shutdown the tomcat server directly. From the messages.log, I can find many many Injected two phase commit, *Can u tell me what cause this and how to prevent it?* Thu Mar 10 09:47:47 CST 2011: Opened [C:\home\heartwater\graph\\nioneo_logical.log.1] clean empty log, version=1 Thu Mar 10 09:47:47 CST 2011: Opened [C:\home\heartwater\graph\/lucene/lucene.log.1] clean empty log, version=0 Thu Mar 10 09:47:47 CST 2011: Opened [C:\home\heartwater\graph\/lucene-fulltext/lucene.log.1] clean empty log, version=0 Thu Mar 10 09:47:47 CST 2011: Non clean shutdown detected on log [C:\home\heartwater\graph\index/lucene.log.1]. Recovery started ... Thu Mar 10 09:47:47 CST 2011: [C:\home\heartwater\graph\index/lucene.log.1] logVersion=0 with committed tx=1 Thu Mar 10 09:47:48 CST 2011: Injected two phase commit, txId=2 Thu Mar 10 09:47:48 CST 2011: Injected two phase commit, txId=3 Thu Mar 10 09:47:48 CST 2011: Injected two phase commit, txId=4 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=5 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=6 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=7 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=8 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=9 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=10 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=11 Thu Mar 10 09:47:49 CST 2011: Injected two phase commit, txId=12 Thu Mar 10 09:47:50 CST 2011: Injected two phase commit, txId=13 Thu Mar 10 09:47:50 CST 2011: Injected two phase commit, txId=14 ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- David Montag Neo Technology, www.neotechnology.com Cell: 650.556.4411 david.mon...@neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Why node's index entry can not be removed from lucene index after the node is deleted from the graph
Thanks Guys~ 2010/12/15 Peter Neubauer peter.neuba...@neotechnology.com Luanne, Mattias, I created a ticket for this, so we don't loose it :) https://trac.neo4j.org/ticket/301 Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Wed, Dec 15, 2010 at 9:24 AM, Mattias Persson matt...@neotechnology.com wrote: It *would* be nice however, to have a method on the Index API which just accepts a Node, and deletes the Lucene document (so that we don't have to iterate through all properties and delete). I'm sure something like this will be showing up in one way or another rather soon. -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Why node's index entry can not be removed from lucene index after the node is deleted from the graph
Thanks Luanne, My case: I indexed several properties on one node, but I missed to remove one property(say, property A) before node.delete(). In such case, seems the index document will not deleted correctly. Although the original document is marked deleted in Luke, but it is strange that a new document with the property A is created using the same _id_. I further encountered a transaction exception, When this node id is re-assigned to a new node.. So the correct way to delete a node should be removing all the index before invoking delete(). Right? 2010/12/14 Luanne Misquitta lmisqui...@saba.com That's strange. I indexed a single property on two nodes, and then used remove method on the index to remove that property- then checked in Luke, and the documents appear to be there, but marked deleted. I could undelete them also via Luke. When I did a Luke-Optimize index, then the documents were well and truly deleted. Not sure what the intended behavior of the index remove is now that you bring this up. Regards Luanne M. Tech Lead twitter / @luannem linkedin / http://in.linkedin.com/in/luannemisquitta skype / luanne.misquitta blog / http://thought-bytes.blogspot.com/ Saba. Power Up Your People. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Samuel Feng Sent: Tuesday, December 14, 2010 12:31 PM To: Neo4j user discussions Subject: Re: [Neo4j] Why node's index entry can not be removed from lucene index after the node is deleted from the graph I checked the index.remove(), it can only delete the key filed from the node entry in index. (The node entry is a document in lucene index) The node entry is still there 2010/12/14 Luanne Misquitta lmisqui...@saba.com You could perhaps also delete the data from the index using: Index.remove(T entity, String key, Object value) However, I assume that if you've added x properties of the node to the index, you'd have to remove all those X properties? Regards Luanne M. Tech Lead twitter / @luannem linkedin / http://in.linkedin.com/in/luannemisquitta skype / luanne.misquitta blog / http://thought-bytes.blogspot.com/ Saba. Power Up Your People. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Samuel Feng Sent: Tuesday, December 14, 2010 12:11 PM To: Neo4j user discussions Subject: [Neo4j] Why node's index entry can not be removed from lucene index after the node is deleted from the graph Dears, IndexNode persons = graphDb.index().forNodes( persons ); Node firstPerson = graphDb.createNode(); Node secondPerson = graphDb.createNode(); persons.add( firstPerson, name, Mattias Persson ); Now I have to delete the firstPerson node from the graph. i.e firstPerson.delete() After the transaction committed, the node is deleted from the graph. However, I then further check the lucene index using LUKE, still can find the node's entry in it. I understand that this node entry will not affect the query result of neo4j. But If my graph has many deletions, there will produce a lot of obsoleted node entry in lucene index. Do u have any ideas that how I can delete this obsoleted index entries or neo4j has some other mechanism to do this? ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Questions on custom query using the index framework
Dear Mattias,
I have updated to M05, All the enhancements work ok.
Thanks a lot for the great job.
2010/11/21 Mattias Persson matt...@neotechnology.com
I've done some stuff regarding this:
- Added similarity configuration parameter (although not very tested
since I don't really know how lucene similarity works).
- Added QueryContext#topDocs(int) which you can set so that only top N
hits are retrieved instead of all (may or may not be a performance
improvement in some scenarios).
- Added IndexHits#currentScore() so that lucene scores are exposed
(although not stuff that's added, but not yet committed, in the current
neo4j transaction if such is open).
I also found some minor performance improvements while doing all this. It'd
be wonderful to have you try this out and tell about how it works out.
These changes will be in the next milestone M05, but you can get latest
snapshots, as we speak, from our repositories.
2010/11/19 Samuel Feng okos...@gmail.com
I think it is ok. It is good no need to use a Similarity instance for
each
query.
2010/11/19 Mattias Persson matt...@neotechnology.com
There's also a problem going with the QueryContext approach... an
IndexSearcher can be shared by multiple threads and multiple queries so
using IndexSearcher#setSimilarity can't be done on a per-query basis.
Setting a similarity in one query may also set it for another
thread/query
which might not want it. So maybe the only option is to go with the
configuration parameter, so that all IndexWriters and IndexSearchers
for
that index gets that Similarity instance set.
WDYT?
2010/11/18 Mattias Persson matt...@neotechnology.com
Allright, then there probably have to be a configuration parameter
for
similarity, and maybe something in QueryContext as well (if you
wouldn't
use
the configuration parameter).
2010/11/18 Samuel Feng okos...@gmail.com
From
http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html
http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html
If
the custom Similarity overwrite the
*lengthNorm
http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html#lengthNorm(java.lang.String
http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html#lengthNorm%28java.lang.String
http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html#lengthNorm%28java.lang.String
,
int)*(field)
http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html#lengthNorm(java.lang.String
http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html#lengthNorm%28java.lang.String
http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html#lengthNorm%28java.lang.String
,
int) method of DefaultSimilarity, need to set the Similarity during
writing
index.
2010/11/19 Mattias Persson matt...@neotechnology.com
2010/11/18 Samuel Feng okos...@gmail.com
Hi,
To my understanding, if you use one
Similarity in indexWriter, you should also use it in
indexSearcher.
And the scoring is not written during writing index, as it
depends
on
the
relationship between query terms and the indexed documents. It
is
calculated
for each query.
But is it a benefit to use it in writer and/or searcher? Is the
similarity
needed at all during writing?
2010/11/18 Mattias Persson matt...@neotechnology.com
Hi,
I am new to neo4j.
I am using a custom analyzer in index in order to
support
Chinese.
chnIndex =
this.graphDb.index().forNodes(product_chinese,
MapUtil.stringMap(analyzer,
org.wltea.analyzer.lucene.IKAnalyzer));
Below is the usage of the IKAnalyzer in lucence.
IndexSearcher isearcher = new
IndexSearcher(directory);
isearcher.setSimilarity(new IKSimilarity()); //
A
custom
Similarity
String word = OLYMPUS 奥林巴斯 E-PL1 时尚白 微型4/3系统相机 套机
含14-42mm镜头;
Query query = IKQueryParser.parse(fieldName, word
);
//
A
custom
QueryParser
TopDocs topDocs = isearcher.search(query, 5); //
Got
the
top
5
System.out.println(Found + topDocs.totalHits);
ScoreDoc[] scoreDocs = topDocs.scoreDocs;
for (int i = 0; i topDocs.totalHits; i++) {
System.out.println(scoreDocs[i].score); // Got
the
score
of
each
matched doc
Document targetDoc =
isearcher.doc
[Neo4j] Why node's index entry can not be removed from lucene index after the node is deleted from the graph
Dears, IndexNode persons = graphDb.index().forNodes( persons ); Node firstPerson = graphDb.createNode(); Node secondPerson = graphDb.createNode(); persons.add( firstPerson, name, Mattias Persson ); Now I have to delete the firstPerson node from the graph. i.e firstPerson.delete() After the transaction committed, the node is deleted from the graph. However, I then further check the lucene index using LUKE, still can find the node's entry in it. I understand that this node entry will not affect the query result of neo4j. But If my graph has many deletions, there will produce a lot of obsoleted node entry in lucene index. Do u have any ideas that how I can delete this obsoleted index entries or neo4j has some other mechanism to do this? ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Why node's index entry can not be removed from lucene index after the node is deleted from the graph
I checked the index.remove(), it can only delete the key filed from the node entry in index. (The node entry is a document in lucene index) The node entry is still there 2010/12/14 Luanne Misquitta lmisqui...@saba.com You could perhaps also delete the data from the index using: Index.remove(T entity, String key, Object value) However, I assume that if you've added x properties of the node to the index, you'd have to remove all those X properties? Regards Luanne M. Tech Lead twitter / @luannem linkedin / http://in.linkedin.com/in/luannemisquitta skype / luanne.misquitta blog / http://thought-bytes.blogspot.com/ Saba. Power Up Your People. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Samuel Feng Sent: Tuesday, December 14, 2010 12:11 PM To: Neo4j user discussions Subject: [Neo4j] Why node's index entry can not be removed from lucene index after the node is deleted from the graph Dears, IndexNode persons = graphDb.index().forNodes( persons ); Node firstPerson = graphDb.createNode(); Node secondPerson = graphDb.createNode(); persons.add( firstPerson, name, Mattias Persson ); Now I have to delete the firstPerson node from the graph. i.e firstPerson.delete() After the transaction committed, the node is deleted from the graph. However, I then further check the lucene index using LUKE, still can find the node's entry in it. I understand that this node entry will not affect the query result of neo4j. But If my graph has many deletions, there will produce a lot of obsoleted node entry in lucene index. Do u have any ideas that how I can delete this obsoleted index entries or neo4j has some other mechanism to do this? ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Questions on custom query using the index framework
I think it is ok. It is good no need to use a Similarity instance for each
query.
2010/11/19 Mattias Persson matt...@neotechnology.com
There's also a problem going with the QueryContext approach... an
IndexSearcher can be shared by multiple threads and multiple queries so
using IndexSearcher#setSimilarity can't be done on a per-query basis.
Setting a similarity in one query may also set it for another thread/query
which might not want it. So maybe the only option is to go with the
configuration parameter, so that all IndexWriters and IndexSearchers for
that index gets that Similarity instance set.
WDYT?
2010/11/18 Mattias Persson matt...@neotechnology.com
Allright, then there probably have to be a configuration parameter for
similarity, and maybe something in QueryContext as well (if you wouldn't
use
the configuration parameter).
2010/11/18 Samuel Feng okos...@gmail.com
From
http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html
http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html
If
the custom Similarity overwrite the
*lengthNorm
http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html#lengthNorm(java.lang.String
http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html#lengthNorm%28java.lang.String
,
int)*(field)
http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html#lengthNorm(java.lang.String
http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html#lengthNorm%28java.lang.String
,
int) method of DefaultSimilarity, need to set the Similarity during
writing
index.
2010/11/19 Mattias Persson matt...@neotechnology.com
2010/11/18 Samuel Feng okos...@gmail.com
Hi,
To my understanding, if you use one
Similarity in indexWriter, you should also use it in indexSearcher.
And the scoring is not written during writing index, as it depends
on
the
relationship between query terms and the indexed documents. It is
calculated
for each query.
But is it a benefit to use it in writer and/or searcher? Is the
similarity
needed at all during writing?
2010/11/18 Mattias Persson matt...@neotechnology.com
Hi,
I am new to neo4j.
I am using a custom analyzer in index in order to support
Chinese.
chnIndex =
this.graphDb.index().forNodes(product_chinese,
MapUtil.stringMap(analyzer,
org.wltea.analyzer.lucene.IKAnalyzer));
Below is the usage of the IKAnalyzer in lucence.
IndexSearcher isearcher = new
IndexSearcher(directory);
isearcher.setSimilarity(new IKSimilarity()); // A
custom
Similarity
String word = OLYMPUS 奥林巴斯 E-PL1 时尚白 微型4/3系统相机 套机
含14-42mm镜头;
Query query = IKQueryParser.parse(fieldName, word );
//
A
custom
QueryParser
TopDocs topDocs = isearcher.search(query, 5); // Got
the
top
5
System.out.println(Found + topDocs.totalHits);
ScoreDoc[] scoreDocs = topDocs.scoreDocs;
for (int i = 0; i topDocs.totalHits; i++) {
System.out.println(scoreDocs[i].score); // Got the
score
of
each
matched doc
Document targetDoc =
isearcher.doc(scoreDocs[i].doc);
System.out.println(Document content is +
targetDoc.toString());
}
How can I use the query method of neo4j's indexing to
implement
1) Add my custom Similartiy implemtation
Btw, what's the difference between setting Similarity on
IndexWriter
vs.
IndexSearcher? Is it performance maybe, that if it's set on the
IndexWriter
then the scoring is written during writes, but if it's set on
IndexSearcher
it's calculated for each query?
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
--
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
--
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
--
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
Re: [Neo4j] Questions on custom query using the index framework
Hi,
To my understanding, if you use one
Similarity in indexWriter, you should also use it in indexSearcher.
And the scoring is not written during writing index, as it depends on the
relationship between query terms and the indexed documents. It is calculated
for each query.
2010/11/18 Mattias Persson matt...@neotechnology.com
Hi,
I am new to neo4j.
I am using a custom analyzer in index in order to support Chinese.
chnIndex = this.graphDb.index().forNodes(product_chinese,
MapUtil.stringMap(analyzer,
org.wltea.analyzer.lucene.IKAnalyzer));
Below is the usage of the IKAnalyzer in lucence.
IndexSearcher isearcher = new IndexSearcher(directory);
isearcher.setSimilarity(new IKSimilarity()); // A custom
Similarity
String word = OLYMPUS 奥林巴斯 E-PL1 时尚白 微型4/3系统相机 套机 含14-42mm镜头;
Query query = IKQueryParser.parse(fieldName, word ); // A
custom
QueryParser
TopDocs topDocs = isearcher.search(query, 5); // Got the top 5
System.out.println(Found + topDocs.totalHits);
ScoreDoc[] scoreDocs = topDocs.scoreDocs;
for (int i = 0; i topDocs.totalHits; i++) {
System.out.println(scoreDocs[i].score); // Got the score of
each
matched doc
Document targetDoc = isearcher.doc(scoreDocs[i].doc);
System.out.println(Document content is +
targetDoc.toString());
}
How can I use the query method of neo4j's indexing to implement
1) Add my custom Similartiy implemtation
Btw, what's the difference between setting Similarity on IndexWriter vs.
IndexSearcher? Is it performance maybe, that if it's set on the IndexWriter
then the scoring is written during writes, but if it's set on IndexSearcher
it's calculated for each query?
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Questions on custom query using the index framework
From
http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html
http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.htmlIf
the custom Similarity overwrite the
*lengthNormhttp://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html#lengthNorm(java.lang.String,
int)*(field)http://lucene.apache.org/java/2_4_0/api/org/apache/lucene/search/Similarity.html#lengthNorm(java.lang.String,
int) method of DefaultSimilarity, need to set the Similarity during writing
index.
2010/11/19 Mattias Persson matt...@neotechnology.com
2010/11/18 Samuel Feng okos...@gmail.com
Hi,
To my understanding, if you use one
Similarity in indexWriter, you should also use it in indexSearcher.
And the scoring is not written during writing index, as it depends on the
relationship between query terms and the indexed documents. It is
calculated
for each query.
But is it a benefit to use it in writer and/or searcher? Is the similarity
needed at all during writing?
2010/11/18 Mattias Persson matt...@neotechnology.com
Hi,
I am new to neo4j.
I am using a custom analyzer in index in order to support
Chinese.
chnIndex = this.graphDb.index().forNodes(product_chinese,
MapUtil.stringMap(analyzer,
org.wltea.analyzer.lucene.IKAnalyzer));
Below is the usage of the IKAnalyzer in lucence.
IndexSearcher isearcher = new IndexSearcher(directory);
isearcher.setSimilarity(new IKSimilarity()); // A custom
Similarity
String word = OLYMPUS 奥林巴斯 E-PL1 时尚白 微型4/3系统相机 套机
含14-42mm镜头;
Query query = IKQueryParser.parse(fieldName, word ); // A
custom
QueryParser
TopDocs topDocs = isearcher.search(query, 5); // Got the
top
5
System.out.println(Found + topDocs.totalHits);
ScoreDoc[] scoreDocs = topDocs.scoreDocs;
for (int i = 0; i topDocs.totalHits; i++) {
System.out.println(scoreDocs[i].score); // Got the score
of
each
matched doc
Document targetDoc = isearcher.doc(scoreDocs[i].doc);
System.out.println(Document content is +
targetDoc.toString());
}
How can I use the query method of neo4j's indexing to implement
1) Add my custom Similartiy implemtation
Btw, what's the difference between setting Similarity on IndexWriter
vs.
IndexSearcher? Is it performance maybe, that if it's set on the
IndexWriter
then the scoring is written during writes, but if it's set on
IndexSearcher
it's calculated for each query?
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
--
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Questions on custom query using the index framework
Mattias,
Thanks a lot. Please find my update in red.
Thanks and Regards,
Samuel
2010/11/14 Mattias Persson matt...@neotechnology.com
2010/11/13 Samuel Feng okos...@gmail.com
Hi,
I am new to neo4j.
I am using a custom analyzer in index in order to support Chinese.
chnIndex = this.graphDb.index().forNodes(product_chinese,
MapUtil.stringMap(analyzer,
org.wltea.analyzer.lucene.IKAnalyzer));
Below is the usage of the IKAnalyzer in lucence.
IndexSearcher isearcher = new IndexSearcher(directory);
isearcher.setSimilarity(new IKSimilarity()); // A custom
Similarity
String word = OLYMPUS 奥林巴斯 E-PL1 时尚白 微型4/3系统相机 套机 含14-42mm镜头;
Query query = IKQueryParser.parse(fieldName, word ); // A custom
QueryParser
TopDocs topDocs = isearcher.search(query, 5); // Got the top 5
System.out.println(Found + topDocs.totalHits);
ScoreDoc[] scoreDocs = topDocs.scoreDocs;
for (int i = 0; i topDocs.totalHits; i++) {
System.out.println(scoreDocs[i].score); // Got the score of each
matched doc
Document targetDoc = isearcher.doc(scoreDocs[i].doc);
System.out.println(Document content is + targetDoc.toString());
}
How can I use the query method of neo4j's indexing to implement
1) Add my custom Similartiy implemtation
That would require new functionality, probably exposed as a configuration
parameter similarity:org.wltea.analyzer.lucene.IKAnalyzer or something.
Maybe QueryContext can also be used here?
2) The TopDocs interface can replaced by the iterator of IndexHits,
right?
In a way, yes. Actually IndexHits wraps a Hits instance (from lucene 2.x),
since it gives all the hits back. To make use of TopDocs instead a method
could be exposed via the QueryContext class. That's definately possible.
3) Possible to get the score for each node in the IndexHits?
Would it be ok with an IndexHits#currentScore() method which could return
the score of the most recently returned node from the IndexHits object?
That is a good idea. Please let me know the schedule of implement this, as
my team is looking forward to this feature. :)
Thank you for the help and time spent on this.
These are great ideas and I'd be happy to try and implement them.
Samuel
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
--
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Questions on custom query using the index framework
Hi,
I am new to neo4j.
I am using a custom analyzer in index in order to support Chinese.
chnIndex = this.graphDb.index().forNodes(product_chinese,
MapUtil.stringMap(analyzer,
org.wltea.analyzer.lucene.IKAnalyzer));
Below is the usage of the IKAnalyzer in lucence.
IndexSearcher isearcher = new IndexSearcher(directory);
isearcher.setSimilarity(new IKSimilarity()); // A custom Similarity
String word = OLYMPUS 奥林巴斯 E-PL1 时尚白 微型4/3系统相机 套机 含14-42mm镜头;
Query query = IKQueryParser.parse(fieldName, word ); // A custom
QueryParser
TopDocs topDocs = isearcher.search(query, 5); // Got the top 5
System.out.println(Found + topDocs.totalHits);
ScoreDoc[] scoreDocs = topDocs.scoreDocs;
for (int i = 0; i topDocs.totalHits; i++) {
System.out.println(scoreDocs[i].score); // Got the score of each
matched doc
Document targetDoc = isearcher.doc(scoreDocs[i].doc);
System.out.println(Document content is + targetDoc.toString());
}
How can I use the query method of neo4j's indexing to implement
1) Add my custom Similartiy implemtation
2) The TopDocs interface can replaced by the iterator of IndexHits, right?
3) Possible to get the score for each node in the IndexHits?
Thank you for the help and time spent on this.
Samuel
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
