Hi Danko, Thank you for your extensive explanation. This makes thinks much cleared for me now.
> For example - a scientific article of 20-30 text pages with lots of terms and > name references could >be a "big document" against an annotation logic that > "knows" those terms. On the other hand a >forum's thread of 100 text pages of > common chatting could be a "piece of cake" for annotation. > From our practical experience so far this amount of statements will be > reached after annotation of >500 000 to 1 000 000 web pages. I think this will absolutely satisfy the needs of my system, so I will keep my work this way and try to adopt your suggestions in the future work. All best, Zoran ----- Original Message ----- From: "DanKo" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]>, "Marin Nozhchev" <[EMAIL PROTECTED]> Cc: "KIM platform discussion" <[email protected]> Date: Wed, 3 Sep 2008 14:27:43 +0300 Subject: Re: [KIM-discussion] Few questions about KIM > Hi Zoran, > > > > Here you are some more details: > > ---- > > About "big document" - this is a document which yields big number of > annotations with a lot of features. > > This depends from one side on the size and information richness of the > document and from other side depends on the complexity of the annotation > logic. > > For example - a scientific article of 20-30 text pages with lots of terms and > name references could be a "big document" against an annotation logic that > "knows" those terms. On the other hand a forum's thread of 100 text pages of > common chatting could be a "piece of cake" for annotation. > > To get a more precise estimation you should do test-annotation. Take several > of the biggest web-pages you could extract and merge them together. Then try > to annotate this Big web-page. If annotation passes without "out-of-memory" > exceptions - then the size of your input is not a problem. You should repeat > this test later if you increase the complexity of the annotation. > > ---- > > Some problems could spring out from this part: > > "... and put those information to the java object model. After all pages > from repository are annotated I transfer those information from java object > model to remote Sesame repository" > > > > You should break the annotation process in small adjustable chunks of pages > to protect from memory shortage. > > ---- > > In any point of the productive life of your system you could switch the KIM > Server to use a stand-alone semantic repository over the same semantic data. > > The downloadable version of the KIM Platform is coupled with a SwiftOWLIM > implementation of the Sesame repository (see: > http://www.ontotext.com/owlim/index.html). > > This Sesame implementation stores a copy of all of the data in the memory. > Let us suppose we have a stand-alone semantic repository configured with the > maximum allowed memory. > > Taking into account the specifics of this Sesame implementation and the > memory limits of 32-bit OS - you can count on storing up to 5 000 000 > statements (or triples). > > From our practical experience so far this amount of statements will be > reached after annotation of 500 000 to 1 000 000 web pages. > > > > If this is quite reachable for the life span of your system - then you should > consider using the BigOWLIM implementation of Sesame (see the same page). > > ---- > > About the synchronization problems - they can occur when you write the data > to the remote semantic repository and someone reads the partially written > data. You will consider if it is substantial for your use-case. > > > HTH > DanKo, > senior developer - KIM Platform > > ----- Original Message ----- > From: Zoran Jeremic > To: Marin Nozhchev ; DanKo > Cc: KIM platform discussion > Sent: Tuesday, September 02, 2008 9:38 PM > Subject: Re: [KIM-discussion] Few questions about KIM > > > Hi Danko, > > > >If I get you right - you currently use the KIM server only as Annotation > >engine with an >embedded Sesame repository and then you transfer the results > >of the annotation to >another external repository. > > > > Yes, that's the way I have integrated KIM in my application. > > > > >the memory usage could be substantial if you annotate very big > >documents/pages. It also >will increase with time when more data is stored > >in the semantic repository. > > > > I'm not sure what you are considering as big documents. My idea is to > annotate several repositories (web sites) of design patterns. I have created > Web crawler that first get all links within one repository that could have > 500-1000 web pages. Then I annotate one page at the time, and put those > information to the java object model. After all pages from repository are > annotated I transfer those information from java object model to remote > Sesame repository. Annotation of these resources should be performed > periodically (once or twice a months) by administrator, in order to reflect > the changes in web repository. It does not have parallel query activity by > that user, but as it is web application it is possible that some other > user(s) do it (aplication uses these information from remote Sesame activity > during the session). I will extend this with annotation of the forum and chat > messages..., that will be performed more often but it will annotate much smaller content. Currently in my research I'm using 32-bit OS and remote Sesame that resides on the same PC. I have created only 23 instances of the 1 class in domain ontology. > > > > Do you believe that this approach could work for me, and is it possible to > keep working this way, and later if I have problems with performance is it > possible to change it and use remote repository? Does the number of instances > in wkb affects the performances? > > > > > You also must have in mind the possibility for synchronization issues. > > > > I didn't understand what do you mean by "synchronization issues" > > > > Thank you for your help > > All best, > > Zoran > > > > --- On Tue, 9/2/08, DanKo <[EMAIL PROTECTED]> wrote: > From: DanKo <[EMAIL PROTECTED]> > Subject: Re: [KIM-discussion] Few questions about KIM > To: [EMAIL PROTECTED], "Marin Nozhchev" <[EMAIL PROTECTED]> > Cc: "KIM platform discussion" <[email protected]> > Date: Tuesday, September 2, 2008, 2:11 AM > > Hi Zoran, > > > > If I get you right - you currently use the KIM server only as Annotation > engine with an embedded Sesame repository and then you transfer the results > of the annotation to another external repository. > > > > If that's o.k. here are my notes about the scale/productivity: > > - Embedded Sesame repository and the memory issue - the memory usage could > be substantial if you annotate very big documents/pages. It also will > increase with time when more data is stored in the semantic repository. If > you run the KIM server on 32-bit OS then there is a limit to the amount of > memory that Java engine could allocate (around 1.4 GB). A way to work around > the memory shortage is to switch to 64-bit OS if possible. Other way you > should use the remote Sesame repository configuration. > > - Embedded Sesame repository and the productivity issue - the productivity > of the annotation is not substantially affected from embedding if you do not > have a parallel query activity by that time. > > - Transferring data between semantic repositories - When you deal with > distributed system you have to have in mind the costs of the data > querying/transformation/transfer and storage to the remote Sesame. You also > must have in mind the possibility for synchronization issues. > > > > Finally - if it's possible - keep it simple! > > > Greetings! > DanKo, > senior developer - KIM Platform > > ----- Original Message ----- > From: Zoran Jeremic > To: Marin Nozhchev ; DanKo > Cc: KIM platform discussion > Sent: Monday, September 01, 2008 6:46 PM > Subject: Re: [KIM-discussion] Few questions about KIM > > > Hi Danko, > > Thank you for your will to help me about this problem. > > > >The problem about starting KIM server with remote Sesame repository may be > >already >history for you but I tried to reproduce it. > > I didn't solve this problem yet. After, a several days of trying to resolve > this, I have applied another approach. I'm using embeded Sesame repository to > annotate web resources, and after I annotate the whole web site I use my > remote Sesame to store the information I need. I need to use my remote Sesame > server because it integrates KIM and several other systems that also use the > same repository. This way, I have cleaner information from KIM, all > statements are created according to my needs and I don't have some > unnecessary information that KIM creates according to Proton ontology. It > works fine, but I'm interesting what do you think about that approach, > specially if it can affect system performance on large scale. > > If this solution is OK, than I don't need to use remote Sesame for KIM > platform. > > Thanks > > Zoran > > --- On Mon, 9/1/08, DanKo <[EMAIL PROTECTED]> wrote: > From: DanKo <[EMAIL PROTECTED]> > Subject: Re: [KIM-discussion] Few questions about KIM > To: [EMAIL PROTECTED], "Marin Nozhchev" <[EMAIL PROTECTED]> > Cc: "KIM platform discussion" <[email protected]> > Date: Monday, September 1, 2008, 6:03 AM > > Hi Zoran, > The problem about starting KIM server with remote Sesame repository may be > already history for you but I tried to reproduce it. > > The attempt was partial success. The root cause of the problem appears to be > the "rmiregistry" service you mention. > > It is not needed both for Sesame and KIM servers. The creation of an RMI > registry is embedded in both of them. > > Besides the presence of "rmiregistry" service conflicts with KIM RMI > registration. > > Unfortunately I couldn't exactly replicate your exceptions. > > So try to start both servers without starting the "rmiregistry". > > Write back if the problem still persists. > > All the best! > > DanKo, > senior developer - KIM Platform > > ----- Original Message ----- > From: Zoran Jeremic > To: Marin Nozhchev > Cc: KIM platform discussion > Sent: Monday, August 25, 2008 9:28 PM > Subject: Re: [KIM-discussion] Few questions about KIM > > > Hi Marin, > > Thank you for your extensive explanation. I'm using kim-platform-2.3-RC1 > release. > > >It's should be easy to set up any Sesame repository to work with KIM. > >The simplest way is to start a separate Sesame process with Java RMI > >support. That's probably the way your started it before. The relevant > >text in the documentation is here - > >http://ontotext.com/kim/doc/sys-doc/ConfigSemRep.html#toc3 . There you > >will find how you should configure KIM to connect to your Sesame server. > >Note that, in that case,you must use a Sesame 1.2.7 server and not a > >Sesame 2.0 one. > That's exactly what I have tried to do. I was successfull to connect my Java > application with > embeded Sesame repository. However, I have the problem to start KIM with > remote sesame repository. Here is a short review of what I have set. > > #semantic.repository.properties > > com.ontotext.kim.KIMConstants.SESAME_REPOSITORY=depths_beta > com.ontotext.kim.KIMConstants.SESAME_IS_REMOTE = true > com.ontotext.kim.KIMConstants.SESAME_RMI_PORT=1098 > com.ontotext.kim.KIMConstants.SESAME_RMI_HOST=192.168.1.5 > com.ontotext.kim.KIMConstants.SESAME_USER=zoran > com.ontotext.kim.KIMConstants.SESAME_PASS=zoran > com.ontotext.kim.KIMConstants.SESAME_MAX_RESULT_COUNT=200 > com.ontotext.kim.KIMConstants.SESAME_DEFAULT_OFFSET=0 > > # in install.properties > > SESAME_SAIL_TYPE=inmem > com.ontotext.kim.KIMConstants.RMI_HOST=192.168.1.5 > com.ontotext.kim.KIMConstants.RMI_PORT=1099 > > # in config_kim.bat > > set SWAN_RMI_HOST=-D"java.rmi.server.hostname=192.168.1.5" > > # in java.policy > > permission java.net.SocketPermission "*:1024-", "listen, connect, accept"; > > > I'm using Sesame web server 1.2.7 on Windows XP Pro. Both Sesame and KIM > platform are on the same PC.I have turned off firewall. I have started > rmiregistry. > > And the error code that I got when trying to start KIM is: > > C:\kim-platform-2.3-RC1\bin>startKIM.bat > "KIM_HOME" set to "C:\kim-platform-2.3-RC1" > java version "1.6.0_04" > Java(TM) SE Runtime Environment (build 1.6.0_04-b12) > Java HotSpot(TM) Client VM (build 10.0-b19, mixed mode) > "TOMCAT_HOME" set to "C:\tomcat" > : : : : : : : : KIM SERVER START : : : : : : : : > Verifying output directory C:\kim-platform-2.3-RC1\context\default\populated > It is now safe to close the KIM Server. > Unable to construct KIMServiceImpl > java.rmi.RemoteException: Bad connection exception while looking for Sesame > serv > ice! > Probably Sesame is not started.; nested exception is: > java.rmi.ConnectException: Connection refused to host: 192.168.1.5; > nest > ed exception is: > java.net.ConnectException: Connection refused: connect > at > com.ontotext.kim.semanticrepository.RepositoryFactory.getSesameFactor > yByHostAndPort(RepositoryFactory.java:48) > at > com.ontotext.kim.semanticrepository.RepositoryFactory.getSesameFactor > y(RepositoryFactory.java:36) > at > com.ontotext.kim.semanticrepository.RepositoryFactory.reconnect(Repos > itoryFactory.java:80) > at > com.ontotext.kim.semanticrepository.RepositoryFactory.getSemanticRepo > sitoryBySesameFactory(RepositoryFactory.java:72) > at > com.ontotext.kim.semanticrepository.RepositoryFactory.createImpl(Repo > sitoryFactory.java:123) > at > com.ontotext.kim.KIMServiceImpl.getSemanticRepositoryAPI(KIMServiceIm > pl.java:138) > at com.ontotext.kim.KIMServiceImpl.<init>(KIMServiceImpl.java:82) > at com.ontotext.kim.KIMServiceImpl.<init>(KIMServiceImpl.java:69) > at > com.ontotext.kim.FactoryInterfaceImpl.bind(FactoryInterfaceImpl.java: > 54) > at com.ontotext.kim.KIMServer.main(KIMServer.java:23) > Caused by: java.rmi.ConnectException: Connection refused to host: > 192.168.1.5; n > ested exception is: > java.net.ConnectException: Connection refused: connect > at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601) > at > sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198 > ) > at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184) > at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322) > at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) > at java.rmi.Naming.lookup(Naming.java:84) > at > com.ontotext.kim.semanticrepository.RepositoryFactory.getSesameFactor > yByHostAndPort(RepositoryFactory.java:42) > ... 9 more > Caused by: java.net.ConnectException: Connection refused: connect > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333) > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195) > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) > at java.net.Socket.connect(Socket.java:519) > at java.net.Socket.connect(Socket.java:469) > at java.net.Socket.<init>(Socket.java:366) > at java.net.Socket.<init>(Socket.java:180) > at > sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirect > SocketFactory.java:22) > at > sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMaster > SocketFactory.java:128) > at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595) > ... 15 more > Exception in thread "main" com.ontotext.kim.client.KIMRuntimeException: > java.rmi > .RemoteException: Bad connection exception while looking for Sesame service! > Probably Sesame is not started.; nested exception is: > java.rmi.ConnectException: Connection refused to host: 192.168.1.5; > nest > ed exception is: > java.net.ConnectException: Connection refused: connect > at > com.ontotext.kim.FactoryInterfaceImpl.bind(FactoryInterfaceImpl.java: > 62) > at com.ontotext.kim.KIMServer.main(KIMServer.java:23) > Caused by: java.rmi.RemoteException: Bad connection exception while looking > for > Sesame service! > Probably Sesame is not started.; nested exception is: > java.rmi.ConnectException: Connection refused to host: 192.168.1.5; > nest > ed exception is: > java.net.ConnectException: Connection refused: connect > at > com.ontotext.kim.semanticrepository.RepositoryFactory.getSesameFactor > yByHostAndPort(RepositoryFactory.java:48) > at > com.ontotext.kim.semanticrepository.RepositoryFactory.getSesameFactor > y(RepositoryFactory.java:36) > at > com.ontotext.kim.semanticrepository.RepositoryFactory.reconnect(Repos > itoryFactory.java:80) > at > com.ontotext.kim.semanticrepository.RepositoryFactory.getSemanticRepo > sitoryBySesameFactory(RepositoryFactory.java:72) > at > com.ontotext.kim.semanticrepository.RepositoryFactory.createImpl(Repo > sitoryFactory.java:123) > at > com.ontotext.kim.KIMServiceImpl.getSemanticRepositoryAPI(KIMServiceIm > pl.java:138) > at com.ontotext.kim.KIMServiceImpl.<init>(KIMServiceImpl.java:82) > at com.ontotext.kim.KIMServiceImpl.<init>(KIMServiceImpl.java:69) > at > com.ontotext.kim.FactoryInterfaceImpl.bind(FactoryInterfaceImpl.java: > 54) > ... 1 more > Caused by: java.rmi.ConnectException: Connection refused to host: > 192.168.1.5; n > ested exception is: > java.net.ConnectException: Connection refused: connect > at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601) > at > sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198 > ) > at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184) > at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322) > at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) > at java.rmi.Naming.lookup(Naming.java:84) > at > com.ontotext.kim.semanticrepository.RepositoryFactory.getSesameFactor > yByHostAndPort(RepositoryFactory.java:42) > ... 9 more > Caused by: java.net.ConnectException: Connection refused: connect > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333) > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195) > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) > at java.net.Socket.connect(Socket.java:519) > at java.net.Socket.connect(Socket.java:469) > at java.net.Socket.<init>(Socket.java:366) > at java.net.Socket.<init>(Socket.java:180) > at > sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirect > SocketFactory.java:22) > at > sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMaster > SocketFactory.java:128) > at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595) > ... 15 more > C:\kim-platform-2.3-RC1\context\default> > > Thanks, > Zoran > > > --- On Mon, 8/25/08, Marin Nozhchev <[EMAIL PROTECTED]> wrote: > From: Marin Nozhchev <[EMAIL PROTECTED]> > Subject: Re: [KIM-discussion] Few questions about KIM > To: [EMAIL PROTECTED] > Cc: "KIM platform discussion" <[email protected]> > Date: Monday, August 25, 2008, 3:22 AM > > > Hi Zoran, > > Thank you for your interest. First of all, please make sure you use the > latest version version of KIM. It's release candidate has been available > for since July. Although the features you ask about were in the previous > version too, the new one provides more stabilty, other features and much > more detailed documentation. You can use your previous download link and > you can request another one from > http://ontotext.com/kim/KIM-downloads.html . > > In particular, here - > http://ontotext.com/kim/doc/sys-doc/AnnotateDocumentsAndTextsExamples.html > - is a code example in the new Developer's guide to KIM that shows how > you ca annotate and/or store web pages in KIM using a small amount of > Java code. It's not currently possible for the release KIM to crawl a > web site and automatically annotate all pages. We have internal tools > that do that, so if you need more help in that aspect, contact us > describing your setup and goal. > > It's should be easy to set up any Sesame repository to work with KIM. > The simplest way is to start a separate Sesame process with Java RMI > support. That's probably the way your started it before. The relevant > text in the documentation is here - > http://ontotext.com/kim/doc/sys-doc/ConfigSemRep.html#toc3 . There you > will find how you should configure KIM to connect to your Sesame server. > Note that, in that case,you must use a Sesame 1.2.7 server and not a > Sesame 2.0 one. > > The other option is to use the Sesame integrated in KIM. In that case, > you will need to edit the Sesame 1.2.7 configuration file - > KIM/config/sesame.inmem.conf . This option is more complicated because > you will need to add the jars of the SAIL implementation that you use to > the classpath of KIM. If you need more help, write me directly with the > version of Sesame that you are using, attach your Sesame configuration > file - system.conf and describe if the standalone Sesame server option > doesn't fit your requirements. > > Storing specific metadata in a specific ontology is achieved by adding > new annotation rules to the existing set, called a GATE pipeline. We are > currently developing a guide for information extraction developers which > will be useful to you. In the mean time, you can send us details of the > domain and the use case, so we can give specific advice. > > Hope this helps, > Marin Nozhchev > lead developer, KIM Plaftform > > > Zoran Jeremic wrote: > > Hi everybody, > > > > Several months ago I have decided to use KIM platform in my research > > work, but in the meantime I have some other stuff to do, so I have > > started my first steps with KIM yesterday. > > > > However, I there are not all information I need in your documentation > > or I couldn't find it. Can you suggest me some references that could > > be useful for me or explain me some questions bellow. > > > > I want to use KIM for semantic annotation of several public online > > resource. From your documents I found instructions how to populate > > documents that are local. Is it possible to populate some online web > > pages, if for example I give some starting address or something like that? > > > > In my application I have already used Sesame repository (database > > based). How can I make KIM to store metadata according to my domain > > ontology in the same Sesame repository? I'm familiar with extending > > Proton ontology with domain ontology, and I have already done it. > > > > Thanks, > > Zoran > > > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > NOTE: Please REPLY TO ALL to ensure that your reply reaches all members of > this mailing list. > > > > KIM-discussion mailing list > > [email protected] > > http://ontotext.com/mailman/listinfo/kim-discussion_ontotext.com > > > > > > _______________________________________________ > NOTE: Please REPLY TO ALL to ensure that your reply reaches all members of > this mailing list. > > KIM-discussion mailing list > [email protected] > http://ontotext.com/mailman/listinfo/kim-discussion_ontotext.com > > > > __________ NOD32 3384 (20080825) Information __________ > > This message was checked by NOD32 antivirus system. > http://www.eset.com > > > _______________________________________________ NOTE: Please REPLY TO ALL to ensure that your reply reaches all members of this mailing list. KIM-discussion mailing list [email protected] http://ontotext.com/mailman/listinfo/kim-discussion_ontotext.com
