Hi Danko, >If I get you right - you currently use the KIM server only as Annotation >engine with an >embedded Sesame repository and then you transfer the results >of the annotation to >another external repository. Yes, that's the way I have integrated KIM in my application. >the memory usage could be substantial if you annotate very big >documents/pages. It also >will increase with time when more data is stored in >the semantic repository. I'm not sure what you are considering as big documents. My idea is to annotate several repositories (web sites) of design patterns. I have created Web crawler that first get all links within one repository that could have 500-1000 web pages. Then I annotate one page at the time, and put those information to the java object model. After all pages from repository are annotated I transfer those information from java object model to remote Sesame repository. Annotation of these resources should be performed periodically (once or twice a months) by administrator, in order to reflect the changes in web repository. It does not have parallel query activity by that user, but as it is web application it is possible that some other user(s) do it (aplication uses these information from remote Sesame activity during the session). I will extend this with annotation of the forum and chat messages..., that will be performed more often but it will annotate much smaller content. Currently in my research I'm using 32-bit OS and remote Sesame that resides on the same PC. I have created only 23 instances of the 1 class in domain ontology. Do you believe that this approach could work for me, and is it possible to keep working this way, and later if I have problems with performance is it possible to change it and use remote repository? Does the number of instances in wkb affects the performances? > You also must have in mind the possibility for synchronization issues. I didn't understand what do you mean by "synchronization issues" Thank you for your help All best, Zoran
--- On Tue, 9/2/08, DanKo <[EMAIL PROTECTED]> wrote: From: DanKo <[EMAIL PROTECTED]> Subject: Re: [KIM-discussion] Few questions about KIM To: [EMAIL PROTECTED], "Marin Nozhchev" <[EMAIL PROTECTED]> Cc: "KIM platform discussion" <[email protected]> Date: Tuesday, September 2, 2008, 2:11 AM Hi Zoran, If I get you right - you currently use the KIM server only as Annotation engine with an embedded Sesame repository and then you transfer the results of the annotation to another external repository. If that's o.k. here are my notes about the scale/productivity: - Embedded Sesame repository and the memory issue - the memory usage could be substantial if you annotate very big documents/pages. It also will increase with time when more data is stored in the semantic repository. If you run the KIM server on 32-bit OS then there is a limit to the amount of memory that Java engine could allocate (around 1.4 GB). A way to work around the memory shortage is to switch to 64-bit OS if possible. Other way you should use the remote Sesame repository configuration. - Embedded Sesame repository and the productivity issue - the productivity of the annotation is not substantially affected from embedding if you do not have a parallel query activity by that time. - Transferring data between semantic repositories - When you deal with distributed system you have to have in mind the costs of the data querying/transformation/transfer and storage to the remote Sesame. You also must have in mind the possibility for synchronization issues. Finally - if it's possible - keep it simple! Greetings! DanKo, senior developer - KIM Platform ----- Original Message ----- From: Zoran Jeremic To: Marin Nozhchev ; DanKo Cc: KIM platform discussion Sent: Monday, September 01, 2008 6:46 PM Subject: Re: [KIM-discussion] Few questions about KIM Hi Danko, Thank you for your will to help me about this problem. >The problem about starting KIM server with remote Sesame repository may be >already >history for you but I tried to reproduce it. I didn't solve this problem yet. After, a several days of trying to resolve this, I have applied another approach. I'm using embeded Sesame repository to annotate web resources, and after I annotate the whole web site I use my remote Sesame to store the information I need. I need to use my remote Sesame server because it integrates KIM and several other systems that also use the same repository. This way, I have cleaner information from KIM, all statements are created according to my needs and I don't have some unnecessary information that KIM creates according to Proton ontology. It works fine, but I'm interesting what do you think about that approach, specially if it can affect system performance on large scale. If this solution is OK, than I don't need to use remote Sesame for KIM platform. Thanks Zoran --- On Mon, 9/1/08, DanKo <[EMAIL PROTECTED]> wrote: From: DanKo <[EMAIL PROTECTED]> Subject: Re: [KIM-discussion] Few questions about KIM To: [EMAIL PROTECTED], "Marin Nozhchev" <[EMAIL PROTECTED]> Cc: "KIM platform discussion" <[email protected]> Date: Monday, September 1, 2008, 6:03 AM Hi Zoran, The problem about starting KIM server with remote Sesame repository may be already history for you but I tried to reproduce it. The attempt was partial success. The root cause of the problem appears to be the "rmiregistry" service you mention. It is not needed both for Sesame and KIM servers. The creation of an RMI registry is embedded in both of them. Besides the presence of "rmiregistry" service conflicts with KIM RMI registration. Unfortunately I couldn't exactly replicate your exceptions. So try to start both servers without starting the "rmiregistry". Write back if the problem still persists. All the best! DanKo, senior developer - KIM Platform ----- Original Message ----- From: Zoran Jeremic To: Marin Nozhchev Cc: KIM platform discussion Sent: Monday, August 25, 2008 9:28 PM Subject: Re: [KIM-discussion] Few questions about KIM Hi Marin, Thank you for your extensive explanation. I'm using kim-platform-2.3-RC1 release. >It's should be easy to set up any Sesame repository to work with KIM. >The simplest way is to start a separate Sesame process with Java RMI >support. That's probably the way your started it before. The relevant >text in the documentation is here - >http://ontotext.com/kim/doc/sys-doc/ConfigSemRep.html#toc3 . There you >will find how you should configure KIM to connect to your Sesame server. >Note that, in that case,you must use a Sesame 1.2.7 server and not a >Sesame 2.0 one. That's exactly what I have tried to do. I was successfull to connect my Java application with embeded Sesame repository. However, I have the problem to start KIM with remote sesame repository. Here is a short review of what I have set. #semantic.repository.properties com.ontotext.kim.KIMConstants.SESAME_REPOSITORY=depths_beta com.ontotext.kim.KIMConstants.SESAME_IS_REMOTE = true com.ontotext.kim.KIMConstants.SESAME_RMI_PORT=1098 com.ontotext.kim.KIMConstants.SESAME_RMI_HOST=192.168.1.5 com.ontotext.kim.KIMConstants.SESAME_USER=zoran com.ontotext.kim.KIMConstants.SESAME_PASS=zoran com.ontotext.kim.KIMConstants.SESAME_MAX_RESULT_COUNT=200 com.ontotext.kim.KIMConstants.SESAME_DEFAULT_OFFSET=0 # in install.properties SESAME_SAIL_TYPE=inmem com.ontotext.kim.KIMConstants.RMI_HOST=192.168.1.5 com.ontotext.kim.KIMConstants.RMI_PORT=1099 # in config_kim.bat set SWAN_RMI_HOST=-D"java.rmi.server.hostname=192.168.1.5" # in java.policy permission java.net.SocketPermission "*:1024-", "listen, connect, accept"; I'm using Sesame web server 1.2.7 on Windows XP Pro. Both Sesame and KIM platform are on the same PC.I have turned off firewall. I have started rmiregistry. And the error code that I got when trying to start KIM is: C:\kim-platform-2.3-RC1\bin>startKIM.bat "KIM_HOME" set to "C:\kim-platform-2.3-RC1" java version "1.6.0_04" Java(TM) SE Runtime Environment (build 1.6.0_04-b12) Java HotSpot(TM) Client VM (build 10.0-b19, mixed mode) "TOMCAT_HOME" set to "C:\tomcat" : : : : : : : : KIM SERVER START : : : : : : : : Verifying output directory C:\kim-platform-2.3-RC1\context\default\populated It is now safe to close the KIM Server. Unable to construct KIMServiceImpl java.rmi.RemoteException: Bad connection exception while looking for Sesame serv ice! Probably Sesame is not started.; nested exception is: java.rmi.ConnectException: Connection refused to host: 192.168.1.5; nest ed exception is: java.net.ConnectException: Connection refused: connect at com.ontotext.kim.semanticrepository.RepositoryFactory.getSesameFactor yByHostAndPort(RepositoryFactory.java:48) at com.ontotext.kim.semanticrepository.RepositoryFactory.getSesameFactor y(RepositoryFactory.java:36) at com.ontotext.kim.semanticrepository.RepositoryFactory.reconnect(Repos itoryFactory.java:80) at com.ontotext.kim.semanticrepository.RepositoryFactory.getSemanticRepo sitoryBySesameFactory(RepositoryFactory.java:72) at com.ontotext.kim.semanticrepository.RepositoryFactory.createImpl(Repo sitoryFactory.java:123) at com.ontotext.kim.KIMServiceImpl.getSemanticRepositoryAPI(KIMServiceIm pl.java:138) at com.ontotext.kim.KIMServiceImpl.<init>(KIMServiceImpl.java:82) at com.ontotext.kim.KIMServiceImpl.<init>(KIMServiceImpl.java:69) at com.ontotext.kim.FactoryInterfaceImpl.bind(FactoryInterfaceImpl.java: 54) at com.ontotext.kim.KIMServer.main(KIMServer.java:23) Caused by: java.rmi.ConnectException: Connection refused to host: 192.168.1..5; n ested exception is: java.net.ConnectException: Connection refused: connect at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601) at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198 ) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184) at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322) at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) at java.rmi.Naming.lookup(Naming.java:84) at com.ontotext.kim.semanticrepository.RepositoryFactory.getSesameFactor yByHostAndPort(RepositoryFactory.java:42) ... 9 more Caused by: java.net.ConnectException: Connection refused: connect at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl..java:333) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:519) at java.net.Socket.connect(Socket.java:469) at java.net.Socket.<init>(Socket.java:366) at java.net.Socket.<init>(Socket.java:180) at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirect SocketFactory.java:22) at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMaster SocketFactory.java:128) at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595) ... 15 more Exception in thread "main" com.ontotext.kim.client.KIMRuntimeException: java.rmi ..RemoteException: Bad connection exception while looking for Sesame service! Probably Sesame is not started.; nested exception is: java.rmi.ConnectException: Connection refused to host: 192.168.1.5; nest ed exception is: java.net.ConnectException: Connection refused: connect at com.ontotext.kim.FactoryInterfaceImpl.bind(FactoryInterfaceImpl.java: 62) at com.ontotext.kim.KIMServer.main(KIMServer.java:23) Caused by: java.rmi.RemoteException: Bad connection exception while looking for Sesame service! Probably Sesame is not started.; nested exception is: java.rmi.ConnectException: Connection refused to host: 192.168.1.5; nest ed exception is: java.net.ConnectException: Connection refused: connect at com.ontotext.kim.semanticrepository.RepositoryFactory.getSesameFactor yByHostAndPort(RepositoryFactory.java:48) at com.ontotext.kim.semanticrepository.RepositoryFactory.getSesameFactor y(RepositoryFactory.java:36) at com.ontotext.kim.semanticrepository.RepositoryFactory.reconnect(Repos itoryFactory.java:80) at com.ontotext.kim.semanticrepository.RepositoryFactory.getSemanticRepo sitoryBySesameFactory(RepositoryFactory.java:72) at com.ontotext.kim.semanticrepository.RepositoryFactory.createImpl(Repo sitoryFactory.java:123) at com.ontotext.kim.KIMServiceImpl.getSemanticRepositoryAPI(KIMServiceIm pl.java:138) at com.ontotext.kim.KIMServiceImpl.<init>(KIMServiceImpl.java:82) at com.ontotext.kim.KIMServiceImpl.<init>(KIMServiceImpl.java:69) at com.ontotext.kim.FactoryInterfaceImpl.bind(FactoryInterfaceImpl.java: 54) ... 1 more Caused by: java.rmi.ConnectException: Connection refused to host: 192.168.1..5; n ested exception is: java.net.ConnectException: Connection refused: connect at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601) at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198 ) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184) at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322) at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) at java.rmi.Naming.lookup(Naming.java:84) at com.ontotext.kim.semanticrepository.RepositoryFactory.getSesameFactor yByHostAndPort(RepositoryFactory.java:42) ... 9 more Caused by: java.net.ConnectException: Connection refused: connect at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl..java:333) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:519) at java.net.Socket.connect(Socket.java:469) at java.net.Socket.<init>(Socket.java:366) at java.net.Socket.<init>(Socket.java:180) at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirect SocketFactory.java:22) at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMaster SocketFactory.java:128) at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595) ... 15 more C:\kim-platform-2.3-RC1\context\default> Thanks, Zoran --- On Mon, 8/25/08, Marin Nozhchev <[EMAIL PROTECTED]> wrote: From: Marin Nozhchev <[EMAIL PROTECTED]> Subject: Re: [KIM-discussion] Few questions about KIM To: [EMAIL PROTECTED] Cc: "KIM platform discussion" <[email protected]> Date: Monday, August 25, 2008, 3:22 AM Hi Zoran, Thank you for your interest. First of all, please make sure you use the latest version version of KIM. It's release candidate has been available for since July. Although the features you ask about were in the previous version too, the new one provides more stabilty, other features and much more detailed documentation. You can use your previous download link and you can request another one from http://ontotext.com/kim/KIM-downloads.html . In particular, here - http://ontotext.com/kim/doc/sys-doc/AnnotateDocumentsAndTextsExamples.html - is a code example in the new Developer's guide to KIM that shows how you ca annotate and/or store web pages in KIM using a small amount of Java code. It's not currently possible for the release KIM to crawl a web site and automatically annotate all pages. We have internal tools that do that, so if you need more help in that aspect, contact us describing your setup and goal. It's should be easy to set up any Sesame repository to work with KIM. The simplest way is to start a separate Sesame process with Java RMI support. That's probably the way your started it before. The relevant text in the documentation is here - http://ontotext.com/kim/doc/sys-doc/ConfigSemRep.html#toc3 . There you will find how you should configure KIM to connect to your Sesame server. Note that, in that case,you must use a Sesame 1.2.7 server and not a Sesame 2.0 one. The other option is to use the Sesame integrated in KIM. In that case, you will need to edit the Sesame 1.2.7 configuration file - KIM/config/sesame.inmem.conf . This option is more complicated because you will need to add the jars of the SAIL implementation that you use to the classpath of KIM. If you need more help, write me directly with the version of Sesame that you are using, attach your Sesame configuration file - system.conf and describe if the standalone Sesame server option doesn't fit your requirements. Storing specific metadata in a specific ontology is achieved by adding new annotation rules to the existing set, called a GATE pipeline. We are currently developing a guide for information extraction developers which will be useful to you. In the mean time, you can send us details of the domain and the use case, so we can give specific advice. Hope this helps, Marin Nozhchev lead developer, KIM Plaftform Zoran Jeremic wrote: > Hi everybody, > > Several months ago I have decided to use KIM platform in my research > work, but in the meantime I have some other stuff to do, so I have > started my first steps with KIM yesterday. > > However, I there are not all information I need in your documentation > or I couldn't find it. Can you suggest me some references that could > be useful for me or explain me some questions bellow. > > I want to use KIM for semantic annotation of several public online > resource. From your documents I found instructions how to populate > documents that are local. Is it possible to populate some online web > pages, if for example I give some starting address or something like that? > > In my application I have already used Sesame repository (database > based). How can I make KIM to store metadata according to my domain > ontology in the same Sesame repository? I'm familiar with extending > Proton ontology with domain ontology, and I have already done it. > > Thanks, > Zoran > > > > ------------------------------------------------------------------------ > > _______________________________________________ > NOTE: Please REPLY TO ALL to ensure that your reply reaches all members of this mailing list. > > KIM-discussion mailing list > [email protected] > http://ontotext.com/mailman/listinfo/kim-discussion_ontotext.com > _______________________________________________ NOTE: Please REPLY TO ALL to ensure that your reply reaches all members of this mailing list. KIM-discussion mailing list [email protected] http://ontotext.com/mailman/listinfo/kim-discussion_ontotext.com __________ NOD32 3384 (20080825) Information __________ This message was checked by NOD32 antivirus system. http://www.eset.com
_______________________________________________ NOTE: Please REPLY TO ALL to ensure that your reply reaches all members of this mailing list. KIM-discussion mailing list [email protected] http://ontotext.com/mailman/listinfo/kim-discussion_ontotext.com
