[Owlim-discussion] Error in predicate statistics - context index related
Hi,
I was able to reproduce problem with error predicate statistics related to
context-index :
Owlim b5849, Sesame 2.6.10, Linux RHEL, Java 7 64 bit, Tomcat 7
1. Edit test.ttl to correctly import foo.owl
2. Create repository :
curl -X POST -H Content-Type:text/turtle -T /home/path/to/test.ttl
localhost:8080/openrdf-sesame/repositories/SYSTEM/rdf-graphs/service?graph=http://www.foo.com/test
3. Run query :
PREFIX rdf:http://www.w3.org/1999/02/22-rdf-syntax-ns#
PREFIX rdfs:http://www.w3.org/2000/01/rdf-schema#
SELECT * WHERE {
GRAPH http://www.ontotext.com/explicit { ?place rdf:type ?placeType .
?placeType rdfs:label ?placeTypeLabel }
} LIMIT 10
4. Look into catalina.out
Best regards,
Marek
foo.owl
Description: Binary data
test.ttl
Description: Binary data
___
Owlim-discussion mailing list
Owlim-discussion@ontotext.com
http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
[Owlim-discussion] Query results in certain situation
Hi everyone,
I experienced strange results in this query :
SELECT * WHERE {
GRAPH http://www.foo.com/one { ?main foo:something ?uri. }
GRAPH http://www.foo.com/two { ?main foo:description ?desc OPTIONAL { ?main
foo:price ?price } }
GRAPH http://www.foo.com/one { OPTIONAL { ?price price:value ?value } ?desc
foo:value ?text . BIND(lang(?text) AS ?language) . } }
BINDINGS ?uri{ ( http://www.uri.com/uri1 ) }
It gives some random results in query answer(only one row should be returned
but hundreds are returned). If I move BIND from GRAPH http://www.foo.com/one
{ ... } clause, and put it right after the GRAPH, it returns correct results.
If I remove OPTIONAL clause from GRAPH http://www.foo.com/one { ... }, the
results are again correct.
Is it correct query?
May I ask whether you made some progress with issues connected to ERROR IN
PREDICATE STATISTICS in these cases? :
1 Testscenario 1.
owlim:enable-context-index true ;
a.During loading there are files in owlim:import which has labels for general
entities. In this case it is label of municipality, but it does not
matter.
b. After creating DB, I load other data, but they are loaded to specific
context. Let's say it is http://www.foo.com/test
c. When you run this query :
SELECT * WHERE {
GRAPH http://www.foo.com/test {
?thing foo:municipality ?municipality
}
?municipality rdfs:label ?label
}
Everything works fine, as it is correct query to make. Data are really in
http://www.foo.com/test and labels which are connected to the
municipality are stored in default graph. No error is in log file.
d.When you run this query, which is incorrect(no results should be returned)
because labels are in default graph, ERROR IN PREDICATE STATISTICS
appears in log file.
SELECT * WHERE {
GRAPH http://www.foo.com/test {
?thing foo:municipality ?municipality .
?municipality rdfs:label ?label
}
}
2. At the same time I think we exeprience the same issue as problem reported
here:
http://www.mail-archive.com/owlim-discussion@ontotext.com/msg02060.html
Best regards,
Marek
___
Owlim-discussion mailing list
Owlim-discussion@ontotext.com
http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
Re: [Owlim-discussion] Owlim-SE not responding with high CPU load
Hi, as long as I understand the error and behaviour it causes (I experienced this error few times before so I'm familiar), it can end with two scenarios, but both are considered as blocker/critical bugs: 1. You didn't recieve results which you should recieve (all data indicate the query is correct but even though you don't get all results you should get) 2. As statistics are broken the query which should normally take 1sec now runs e.g 20 minutes. I think the second option fits to you. When query is executed, it is normally running and in some future it will give results, but as it needs the much higher time to return results it blocks database(instead of taking database resources for 1second it uses it for 20 minutes and therefore you see such high CPU usage). The thing that you noticed this behaviour this morning is just lucky concidence and sooner or later you will certainly fall into trouble. I think the statistics which are broken are always related to specific predicates. As you didn't use the predicate which has broken statistics, you didn't notice it. From my previous experience, disabling context-index + also set index compression to -1 could solve some issues (but probably you'll have to reload the database). It is certainly not cure, but it can help you to work with application until the bug will be fixed. Hope I explain it bit to you. Hope the fix will come soon. Best regards, Marek From: Stefano Parmesan parme...@spaziodati.eu To: Marek marek_su...@yahoo.co.uk Cc: owlim-discussion@ontotext.com owlim-discussion@ontotext.com Sent: Thursday, 28 March 2013, 14:10 Subject: Re: [Owlim-discussion] Owlim-SE not responding with high CPU load $ grep ERROR IN PREDICATE STATISTICS catalina.out | wc -l 32368 (since the 25th) Apparently the last error is of yesterday afternoon, but we experienced such problems this morning as well, I can't say if they are related. Thanks and regards 2013/3/28 Marek marek_su...@yahoo.co.uk we experienced very similar behaviour with nearly the same usecase. i reported one bug which was in our case temporarily solved by turning off context index. recently i found other very similar issue, which is not reportwd yet as i cant figure out the cause. look pls into catalina.out whether there is not log error in predicate statistics which appears in both mentioned issues. maybe we hit the same problem. best regards, marek From: Stefano Parmesan Sent: 28.3.2013 12:09 To: owlim-discussion@ontotext.com Subject: [Owlim-discussion] Owlim-SE not responding with high CPU load Hi everybody, We are evaluating Owlim-SE 5.3.5849 but we are encountering some issues: Our test repository contains around 1 million triples (around 100 of those are owl:sameAs) and have concurrent applications both inserting and querying (through sesame-console and the sparql endpoint provided by the sesame-workbench). The machine is a 12-core 64GB ram debian machine. Everything worked fine as of today, when something happened while we were submitting a high load to the sparql endpoint. Since then, tomcat7 uses from 200% to 650% of cpu, and the sparql endpoint does not respond with even simple queries. We tried restarting tomcat7 multiple times, but as soon as it comes back the CPU usage increases again and there's no way to do anything (through both sesame-workbench and sesame-console). Could this be due to some misconfiguration? Is this a known issue? How can we know what's really happening (apart from checking .aduna/openrdf-sesame/logs)? We could clear the repository and start from scratch, but as we are evaluating Owlim for production usage we need to find out what's the issue to better understand if it fits our needs. -- Dott. Stefano Parmesan Web Developer ~ SpazioDati s.r.l. Via del Brennero, 52 – 38122 Trento – Italy -- Dott. Stefano Parmesan Web Developer ~ SpazioDati s.r.l. Via del Brennero, 52 – 38122 Trento – Italy___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
Re: [Owlim-discussion] Loading a Large Triple Store using OWLIM-SE
Hi, if you want to see progress in loading, there is and option to use standard curl command instead of openrdf-workbench. It gives you some information what is already loaded. To load files into owlim(from .trig file), run this command in your linux shell : curl -X POST -H Content-Type:application/x-trig -T /path/to/data/datafile.trig localhost:8080/openrdf-sesame/repositories/repository-name/statements If you have xml style data, change content type to application/rdf+xml If you load big amount of data, I recommend to use configuration.xls which is part of OWLIM-SE.zip. It can help you to set datastore properly. Hope this will help. Best regards, Marek From: Joshua Greben jgre...@stanford.edu To: owlim-discussion@ontotext.com Sent: Thursday, 28 March 2013, 22:30 Subject: [Owlim-discussion] Loading a Large Triple Store using OWLIM-SE Hello all, I am new to this list and to OWLIM-SE and was wondering if anyone could offer advice for loading a large triple store. I am trying to load 670M triples into a repository using the openrdf-sesame workbench under tomcat6 on a single linux VM with 64-bit hardware and 64GB of memory. My JVM has the following: -Xms32g -Xmx32g -XX:MaxPermSize=256m Here is the log info for my repository configuration: ... [INFO ] 2013-03-27 13:57:00,720 [repositories/BFWorks_STF] Configured parameter 'entity-id-size' to '32' [INFO ] 2013-03-27 13:57:00,720 [repositories/BFWorks_STF] Configured parameter 'enable-context-index' to 'false' [INFO ] 2013-03-27 13:57:00,720 [repositories/BFWorks_STF] Configured parameter 'entity-index-size' to '1' [INFO ] 2013-03-27 13:57:00,720 [repositories/BFWorks_STF] Configured parameter 'tuple-index-memory' to '1600m' [INFO ] 2013-03-27 13:57:00,721 [repositories/BFWorks_STF] Configured parameter 'cache-memory' to '3200m' [INFO ] 2013-03-27 13:57:00,721 [repositories/BFWorks_STF] Cache pages for tuples: 83886 [INFO ] 2013-03-27 13:57:00,721 [repositories/BFWorks_STF] Cache pages for predicates: 0 [INFO ] 2013-03-27 13:57:00,721 [repositories/BFWorks_STF] Configured parameter 'storage-folder' to 'storage' [INFO ] 2013-03-27 13:57:00,741 [repositories/BFWorks_STF] Configured parameter 'in-memory-literal-properties' to 'false' [INFO ] 2013-03-27 13:57:00,742 [repositories/BFWorks_STF] Configured parameter 'repository-type' to 'file-repository' The loading came to a standstill after 19 hours and tomcat threw an OutOfMemoryError: GC overhead limit exceeded. My question is what the application is doing with all this memory and whether I configured my instance correctly for this load to finish. I also see a lot of entries in the main log such as this: [WARN ] 2013-03-28 08:50:59,114 [repositories/BFWorks_STF] [Rio error] Unescaped backslash in: L\'ambassadrice (314764886, -1) Could these Rio errors be contributing to my troubles? I was also wondering if there was a way to configure logging to be able to track the application's progress. Right now these warnings are the only way I can tell how far the loading has progressed. Advice from anyone who has experience successfully loading a large triplestore is much appreciated! Thanks in advance! - Josh Joshua Greben Library Systems Programmer Analyst Stanford University Libraries (650) 714-1937 jgre...@stanford.edu ___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
Re: [Owlim-discussion] [Sesame] Few features request for upcoming 2.7
2. I was not aware that .pie is specific only to OWLIM, therefore my apology. What is then the standard format for rulewriting? What I mean is that in what format are predefined rulesets in Sesame Workbench stores? Main idea was to add custom ruleset in workbench, I agree that editing would not make much sense. 3. I understand that I can't change ruleset just like that... ,my point is that our application is still in developement and therefore I change ruleset from time to time (but I don't expect it is immediatedly applied to repository). I thought OWLIM compiles rules during creating new repository. Therefore I didn't expect changes in .pie are immediatedly used. Our common scenario is that : a. There is working repository with working ruleset. b. I would like to add some rules (but without recomputing them and without having additional rules file) c. If set of new rules will be final, I can drop the working repository and recreate it with new rules. It happened to me few times that I changed .pie file and then Tomcat restart came and I couldn't delete repository nor to create the same one with the new rules. Therefore I had to revert .pie file, delete repository and then I can again change .pie and create new repository. If it has some technical explanation I would accept it and try to avoid the same situations. It just happened to me few times and therefore I asked whether it is possible to change it. Best regards, Marek From: Barry Norton barry.nor...@ontotext.com To: owlim-discussion@ontotext.com Sent: Tuesday, 19 February 2013, 12:24 Subject: Re: [Owlim-discussion] [Sesame] Few features request for upcoming 2.7 2. Since PIE is an OWLIM-specific rule format, it would not make sense to add this to the Sesame Workbench. The OWLIM Workbench already has an option to upload a custom ruleset. While editing is something that could be considered for the OWLIM Workbench, it's probably a feature that's used by only a small proportion of users, for a very small proportion of their time, that other features would be likely to be prioritised. 3. a) it's worth considering that rules are internalised, but this would make it more difficult to change the ruleset used in an existing repository (via SPARQL). Would it not just be easier, when you're using a custom ruleset, to create a separate ruleset file for each repository and a separate one to make edits? In the case of deletion I do agree (on the Sesame list) that having to initialise the repository before deleting it is a pain. Barry On 19/02/13 10:58, Barry Norton wrote: Marek, a number of these are OWLIM requests and I'll answer those separately on the owlim-discussion list... Barry On 19/02/13 10:55, Marek Surek wrote: Hi all, I would like to propose a few new features in upcoming 2.7 release. As I don't know whether 2.7 is not feature freeze for now, I post the features here. 1. Create repository in workbench directly from .ttl file 2. Create/add/edit custom ruleset in workbench (the ability to add custom .pie file would be great) 3. Change the behavior for deletition of repository. The problem is that when I slightly change .pie or remove .pie or .ttl file, I can't delete existing repository. The developers from Ontotext told me it is because Sesame needs to initialize repository first, but the initialization cannot be done. It is possible to change this behaviour in the way : a) the whole ttl and pie information will be stored in repository so there will be no need to have exactly the same files for the whole repository life and repository can be deleted anytime. b) the initalization of repository is not required when delete repository action. Is it possible to consider at least few of this features? Best regards, Marek -- View this message in context: http://sesame-general.435816.n3.nabble.com/Few-features-request-for-upcoming-2-7-tp4025791.html Sent from the sesame-general mailing list archive at Nabble.com. -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_d2d_feb ___ Sesame-general mailing list sesame-gene...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/sesame-general ___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion ___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion___ Owlim-discussion mailing list Owlim-discussion@ontotext.com
Re: [Owlim-discussion] [Sesame] Few features request for upcoming 2.7
As it is so, I will create separate pie file and deploy them when they will be ready. Thank you for your explanation. Marek From: Barry Norton barry.nor...@ontotext.com To: Marek Šurek marek_su...@yahoo.co.uk Cc: owlim-discussion@ontotext.com owlim-discussion@ontotext.com Sent: Tuesday, 19 February 2013, 13:15 Subject: Re: [Owlim-discussion] [Sesame] Few features request for upcoming 2.7 On 19/02/13 11:59, Marek Šurek wrote: 2. I was not aware that .pie is specific only to OWLIM, therefore my apology. What is then the standard format for rulewriting? What I mean is that in what format are predefined rulesets in Sesame Workbench stores? Main idea was to add custom ruleset in workbench, I agree that editing would not make much sense. No worries. There is no standard for Sesame, it doesn't have a general rule engine in any language to replace the rules used - this is one of the advantages of OWLIM. 3. I understand that I can't change ruleset just like that... ,my point is that our application is still in developement and therefore I change ruleset from time to time (but I don't expect it is immediatedly applied to repository). I thought OWLIM compiles rules during creating new repository. Therefore I didn't expect changes in .pie aris e immediatedly used. Our common scenario is that : a. There is working repository with working ruleset. b. I would like to add some rules (but without recomputing them and without having additional rules file) c. If set of new rules will be final, I can drop the working repository and recreate it with new rules. So new rules in the file at which the repository points are applied, after these are compiled (each start up), as new data is added. Note that this is different from re-inferring over the existing data (which is also possible). In order words the ruleset is not fixed for the lifetime of the repository. It happened to me few times that I changed .pie file and then Tomcat restart came and I couldn't delete repository nor to create the same one with the new rules. Therefore I had to revert .pie file, delete repository and then I can again change .pie and create new repository. If it has some technical explanation I would accept it and try to avoid the same situations. It just happened to me few times and therefore I asked whether it is possible to change it. For the reason above (that the rules executed on new data is added are changed according to the file contents) you should not edit the file (rather a copy) unless you're sure those are the rules you want executed. Barry From: Barry Norton barry.nor...@ontotext.com To: owlim-discussion@ontotext.com Sent: Tuesday, 19 February 2013, 12:24 Subject: Re: [Owlim-discussion] [Sesame] Few features request for upcoming 2.7 2. Since PIE is an OWLIM-specific rule format, it would not make sense to add this to the Sesame Workbench. The OWLIM Workbench already has an option to upload a custom ruleset. While editing is something that could be considered for the OWLIM Workbench, it's probably a feature that's used by only a small proportion of users, for a very small proportion of their time, that other features would be likely to be prioritised. 3. a) it's worth considering that rules are internalised, but this would make it more difficult to change the ruleset used in an existing repository (via SPARQL). Would it not just be easier, when you're using a custom ruleset, to create a separate ruleset file for each repository and a separate one to make edits? In the case of deletion I do agree (on the Sesame list) that having to initialise the repository before deleting it is a pain. Barry On 19/02/13 10:58, Barry Norton wrote: Marek, a number of these are OWLIM requests and I'll answer those separately on the owlim-discussion list... Barry On 19/02/13 10:55, Marek Surek wrote: Hi all, I would like to propose a few new features in upcoming 2.7 release. As I don't know whether 2.7 is not feature freeze for now, I post the features here. 1. Create repository in workbench directly from .ttl file 2. Create/add/edit custom ruleset in workbench (the ability to add custom .pie file would be great) 3. Change the behavior for deletition of repository. The problem is that when I slightly change .pie or remove .pie or .ttl file, I can't delete existing repository. The developers from Ontotext told me it is because Sesame needs to initialize repository first, but the initialization cannot be done. It is possible to change this behaviour in the way : a) the whole ttl and pie
[Owlim-discussion] Query strange behavior
Hi everyone,
I experienced strange behavior during query answering, but Im not sure where
can be problem.
This query is not working(no results returned) :
SELECT DISTINCT ?key ?value ?label ?score FROM NAMED
http://www.ontotext.com/explicit { ?value luc:sestateIndex
word*^^xsd:string . ?value luc:score ?score . ?value rdfs:label ?label .
GRAPH http://www.ontotext.com/explicit { ?value rdf:type ?key } . }
but if I remove one of these (no matter which one)
?value rdfs:label ?label . OR GRAPH http://www.ontotext.com/explicit { ?value
rdf:type ?key} OR ?value luc:score ?score
suddelny all is perfectly working as I expect. Lucene index is properly created
by this query :
INSERT DATA
{
luc:index luc:setParam uris .
luc:include luc:setParam literals .
luc:moleculeSize luc:setParam 1 .
luc:includePredicates luc:setParam
http://www.w3.org/2000/01/rdf-schema#label
http://www.w3.org/2004/02/skos/core#altLabel; .
luc:analyzer luc:setParam my.own.analyzer .
luc:sestateIndex luc:createIndex true .
}
Could you guide me to possible solution for this?
Best regards,
Marek
___
Owlim-discussion mailing list
Owlim-discussion@ontotext.com
http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
[Owlim-discussion] Cannot delete repository with changed/removed ruleset
Hi everyone, I experienced one situation which I consider not standard. When I change rules in .pie files or remove the whole .pie file, I cannot DELETE the repository. It ends with following exception : javax.servlet.ServletException: org.openrdf.repository.RepositoryException: Failed to delete repository: org.openrdf.repository.RepositoryException: org.openrdf.sail.SailException: Invalid rule set: /path/to/ruleset/rules.pie (500) I think that deleting repository should be allowed whether .pie file exists or not or is changed. As long as I again don't know whether Sesame doesnt allow it or it should be aimed to OWLIM I send this request here. The only solution is revert .pie file, delete repository. Is it possible to change this behavior? Best regards, Marek ___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
[Owlim-discussion] MALFORMED DATA: An invalid XML character
Hi everyone, I've some strange error in my appliacation. This error didn't occur during loading but during reqular repository usage. May I ask whether the problem is related to Sesame or to Owlim and how can I get rid of it? org.openrdf.repository.RepositoryException: Transaction failed: MALFORMED DATA: An invalid XML character (Unicode: 0x5) was found in the element content of the document. (400) at org.openrdf.http.client.HTTPClient.sendTransaction(HTTPClient.java:738) at org.openrdf.repository.http.HTTPRepositoryConnection.commit(HTTPRepositoryConnection.java:277) Thank you for your answer. Best regards, Marek ___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
[Owlim-discussion] Plugin not found in classpath
Hi, I see this error message for quite long time in our logs, but as it was not crucial I ignored it. May I ask what the error message means and why it's there? What are consequences of such error, because error is quite strong log message? We use geo-spatial plugin and lucene plugin, but the error message is there even when they are disabled. [ERROR] 2013-01-11 10:39:39,120 [repositories/repository-name] Plugin '' not found in classpath. Skipping... Best regards, Marek ___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
Re: [Owlim-discussion] OWLIM 5.3 - SPARQL and math functions
Hi, there are two options, as long as I know. The first one is to try BIND((?a * ?a) AS ?aSquared) The second one could be to implement own Sesame SPARQL function(if you need complex math function as logarithms etc.). It is quite easy to implement and then insert into lib folder in server : http://rivuli-development.com/further-reading/sesame-cookbook/creating-custom-sparql-functions/ Hope this helps. Best regards, Marek From: Fabian Cretton fabian.cret...@hevs.ch To: owlim-discussion@ontotext.com Sent: Wednesday, 21 November 2012, 15:59 Subject: [Owlim-discussion] OWLIM 5.3 - SPARQL and math functions Hi, Is there anyway to use math functions (as square root) in SPARQL queries with OWLIM 5.3 ? If not, is it something planned or not to count on in a near future ? Thanks for the information Fabian ___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
[Owlim-discussion] Bunch of questions and problems
Hi everyone,
during time(summer) I found some problems which I don't fully understand so
here they are :).
I run OWLIM 5.2(b5316) with Sesame 2.6.9 on both Windows 7 64bit(dev. machine)
and Linux(test) server. Some issues are connected to both platforms, some only
to Windows. Both enviroments share these settings. They have differences only
with RAM for cache.
owlim:enablePredicateList false ;
owlim:enable-context-index true ;
owlim:enable-optimization true ;
owlim:enable-literal-index true ;
owlim:predicate-memory 0 ;
owlim:in-memory-literal-properties false;
owlim:fts-memory 0 ;
owlim:index-compression-ratio 30 ;
owlim:ftsIndexPolicy never ;
owlim:journaling true ;
owlim:transaction-mode safe ;
1. On both platforms deleting repository did not remove files connected with
repository. Do you know where can be problem? I know that it was fixed(and it
worked somewhere around 2.6.5-2.6.6 of Sesame, but it is not happening now). I
wrote also to Sesame forum with no response.
2. I would like to ask when or how is OWLIM's built-in cache used, because I
have these kind of experience and I don't know whether is correct. I run some
complex query multiple times and it still takes few seconds(10s) to get
results. Also simple COUNT(*) queries is not cached as time is not smaller
after multiple times runing. The strange thing is that more RAM is taken by
OWLIM so something is happening. There was also problem that whether I gave to
repository 5 or 30GB of RAM(owlim:cache-memory), no performance boost was
visible. I also have to point out that no INSERTs into database was made
between firing queries. Could you guide me where can be problem or how can be
OWLIM's cache invoked?
3. The next problem is connected only to Windows enviroment. After loading data
to repository and doing some SPARQL inferencing data are normally available.
After normal Tomcat shutdown and startup (sometimes problem is visible only
after restaring NB) SPARQL insertions are gone. This is not happening on Linux
machine. As Im using the same processing tool(automatized loading + SPARQL
inferening) for both platforms and share the same repository settings I'm not
sure where can be problem. It seems as if transaction-mode on Windows was
always fast. Am I missing something?
4. Is there any way to quickly count triples (explicit or implicit) in
repository using query? I know about Sesame's
http://localhost:8080/openrdf-sesame/repositories/rep_id/size
option but I need some query approach. The standard COUNT using ?s ?p ?o is on
large datasets taking tens of minutes to proceed and amount of RAM needed is
quite big.
6. I found something what could be some mysterious bug. Here is example
DELETE {
?s predicate:isSomething true^^xsd:boolean
}
INSERT{
?s predicate:isSomething false^^xsd:boolean
}
WHERE {}
on small dataset it works correctly. The problem appears when doing this on
40m+ triples dataset(the ?s applies to 100k+ bindings). Suddenly INSERT part of
query is correctly executed but DELETE is not. Then I have for the same
property true and false values which causes me big trouble. And I also
would like to ask whether some consistency check's can be applied that the
boolean property can't be true and false at the same time.(adding consistency
check in .pie file? or is there something better?)
7. My last question is connected to loading statements also on large datasets.
Around some higher number of triples (40m) is loading performing worse than
before. It could be normal behaviour as more statements are in repository, but
I realized that processor is using during loading only about 10-20%. Could be
this problem of slow disc? Do you thing that SSDs could solve the problem? Or
could have the problem other roots? I load still cca. the same size files in
RDF/XML format(not the best one). As properties are not totally type correct I
can't use batch loading as proposed in some of Jeen's optimization articles.
8. We use different input format (.trig) for backups. Medium sized
dataset(loaded 48x200MB RDF/XML files) backup zipped in single .zip is loaded
significantly faster than loading the files one by one. Is it because of .trig
itself or because there are some optimizations? If .trig causes this much, I
will make some .xml-.trig converter.
Thank you for your time and answer and looking for our cooperation.
Best regards,
Marek___
Owlim-discussion mailing list
Owlim-discussion@ontotext.com
http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
[Owlim-discussion] Backup in running system
Hi, I want to ask how exactly backup is performed on running system. I read backup part in FAQ and everything seems fine, but I'm confused with word 'seamlessly'. Therefore using programatical approach : 1. Is backup consistency-safe on running system? Does it differ using OWLIM-SE or OWLIM-EE? My best guess is whether in OWLIM-EE is backup performed on one working node in following scenario : a. One working node is chosen and it stops to be up-to-date/replicated with other working nodes b. Full backup is made on this working node c. Working node is added back to work/replication and is updated with other nodes 2. Doesn't performing backup on running system degrade performance on such level it is unusable by high number of users? 3. Is there any way of incremental backup? The used store has tens of GB and therefore backupfile size + time needed for backup will be enormous if consistent backup cannot be made on running system. Thank you for your support. Best regards, Marek ___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
Re: [Owlim-discussion] Poor Performance in OWLIM-SE
Hi,
I experience the similar issues. The build b5123(OWLIM 5.0) was last build
which didn't take advantage of new QueryOptimizer and it gave me expected
performance. The new query optimizer(which causes the problems) seems to be
fully functioning from build b5183 (at least it was the point where my query
issues started to appear). I think there won't be much difference between your
version (b5208) and b5123 as long as it suppose to be maintaince release mostly
aimed on bugfixing.
Best regards,
Marek
From: Jeni Tennison j...@jenitennison.com
To: Barry Bishop barry.bis...@ontotext.com
Cc: owlim-discussion@ontotext.com; jeni.tenni...@tso.co.uk;
amna.farh...@tso.co.uk
Sent: Tuesday, 10 July 2012, 23:35
Subject: Re: [Owlim-discussion] Poor Performance in OWLIM-SE
Hi Barry,
Thank you for investigating. That explains a lot of my current performance woes
:)
So long as we know a fix is coming next week, I think we'll be able to cope,
but just to explore other options so that we can present them to our customer:
do you know when the bug was introduced? How many versions back would we have
to revert, while waiting for the next release, to avoid it?
Thanks,
Jeni
On 10 Jul 2012, at 15:49, Barry Bishop wrote:
Hi Jeni,
This is indeed a bug in OWLIM-SE. It is quite specific and occurs only when a
FILTER contains two or more expressions OR'd together and these expressions
contain equalities.
The problem causes the OWLIM specific query optimisation step to abort and
default back to the (unoptimised) Sesame query evaluation steps. This is why
you see a degradation in performance to something worse than OWLIM-Lite
(which uses the Sesame evaluation mechanism, but with optimisation).
We are planning a new version of OWLIM-SE very soon, originally intended to
include Sesame 2.6.7 (which has other query optimisation fixes). However, we
have detected an unrelated bug in the SPARQL parser in this version of Sesame
and so we are working now to release Sesame 2.6.8 at the end of this week,
OWLIM next week.
The question now is, how problematic is this current problem for you? Can you
wait until next week for a new release of OWLIM?
All the best and thanks for reporting this,
barry
On 10/07/12 10:48, Jeni Tennison wrote:
Hi Barry2,
I hadn't, but I just did and it had no discernible effect.
Thanks,
Jeni
On 9 Jul 2012, at 22:26, Barry Norton wrote:
Just to throw in - you've tried this with a UNION rather than the property
path alternative, I guess?
Barry2
On 09/07/2012 21:07, Barry Bishop wrote:
Hi Jeni,
I suspect you have run in to a recent query optimisation problem, but to
be sure, could you tell me which version of OWLIM-SE you are using
(including build number)?
I'll check with the developers in the morning.
Regards,
barry
Barry Bishop
OWLIM Product Manager
Ontotext AD
Tel: +43 650 2000 237
email: barry.bis...@ontotext.com
skype: bazbishop
www.ontotext.com
On 09/07/12 17:31, Jeni Tennison wrote:
Hi,
I have a query which is performing very poorly in OWLIM-SE, but very well
in OWLIM-Lite. This is a simplified version which displays the same
characteristics:
PREFIX task: http://www.legislation.gov.uk/def/task/
SELECT ?process (COUNT(?task) AS ?count)
WHERE {
{
SELECT DISTINCT ?task ?process
WHERE {
?task task:process ?process .
FILTER (
?process =
http://www.legislation.gov.uk/id/process/prepare/effects ||
?process =
http://www.legislation.gov.uk/id/process/review/prepare/effects
)
OPTIONAL { ?task task:endedBy|task:endedAt ?unassignedEnded . }
FILTER (!BOUND(?unassignedEnded))
}
}
}
GROUP BY ?process
This takes several seconds in OWLIM-SE but only about 100ms with
OWLIM-Lite (there are other differences in the servers of the two
repositories, but only ones that would lead me to imagine that the
OWLIM-SE machine would be faster).
To give you an idea of size, the results are:
http://www.legislation.gov.uk/id/process/prepare/effects 906
http://www.legislation.gov.uk/id/process/review/prepare/effects 907
The lines which seem to be causing the problem are
OPTIONAL { ?task task:endedBy|task:endedAt ?unassignedEnded . }
FILTER (!BOUND(?unassignedEnded))
I have tried several rewrites of these lines, including using FILTER (
NOT EXISTS { ... } ). This is the most performant that I've found, but
it's still too slow. If I remove these lines, the results are:
http://www.legislation.gov.uk/id/process/prepare/effects 2594
http://www.legislation.gov.uk/id/process/review/prepare/effects 2594
so it's having to do a bit of filtering, but it's not like there are tens
of millions of tasks for it to exclude.
Does anyone have any suggestions for how I might rewrite the query to get
better performance, or why there might be this big difference in
performance between
Re: [Owlim-discussion] Owlim-lite Inference
Hi Stefano, its certainly possible and this kind of work relies on TRREE reasoner. There is no default ruleset as long as I know(but in openrdf-workbench the choosen ruleset in selectbox is during creating new repository Empty which is not good for you ). I don't know how you've created your repository. Did you use openrdf-workbench or did you create it manually throught openrdf-console? My guess is that you use openrdf-workbench. In that case you should create repository in this way : 1. Deploy openrdf-sesame.war and openrdf-workbench.war (e.g. copy these two files into .../tomcat/webapps/ directory). 2. Run Tomcat 3. Open web browser and go to : http://localhost:8080/openrdf-workbench 4. Go to Create New Repository 5. Choose Owlim DB and write database name 6. Click Next and then you will see Ruleset option with many rules to choose. As Damian said, in your case you should not choose Empty or RDFS ruleset... Let's try OWL2-QL, which definitely contain equivalentClass rule 7. Then you try to do the same thing with this newly created repository and you should recieve results that you expect Best regards, Marek From: Stefano Ortona sedan...@hotmail.it To: damyan dam...@sirma.bg Cc: owlim-discussion@ontotext.com Sent: Friday, 6 July 2012, 16:27 Subject: Re: [Owlim-discussion] Owlim-lite Inference Hello, sorry I'm new and I'm not very familiar with this concept. I didn't set any ruleset, so I'm using the default one. I have the axiom A owl:equivalentClass B and i want to avoid to write B owl:equivalentClass A, but when I query for the equivalent class of B i want also A to be returned, so I need a kind of reasoner. Is that possible and how? Thank you, Stefano On 06/lug/2012, at 15.09, damyan wrote: Hi Steafno, which ruleset you are using because 'rdfs' or 'empty' do not have the necessary rules/axioms to derive such conclusions? If it is other, then you should relate both of your classes with owl:equivalentClass property or alternatively, make them mutually rdfs:subClassOf to each other... HTH, Damyan Ognyanov Ontotext AD On 7/6/2012 4:28 PM, Stefano Ortona wrote: Hello everybody, I successfully instal the owlim-lite server, load my own ontology and now i'd like to query it via sparql http endpoint using the inference. In particular, i have to class in my ontology A and B and i specified A isEquivalentClass of B. At the moment, if a query: give me all the equivalent class of B the class A is not returned, because there's no explicit axiom that says that. How can I set the inference in order to get A as equivalent class of B? Thank you, Stefano Ortona ___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion ___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
Re: [Owlim-discussion] Poor performance for Sparql queries with property path optional elements
Thank you Ruslan, I will revert back to OWLIM 5.0 b5123 using Sesame 2.6.5 where all things seems fines (except the removeRepository, but I can live with it). I was just confused as I made tons of query testing to realise what are performance strong and weak queries and suddenly it had changed in bad way and I was worried about project destiny. Good luck with bugfixing and looking forward for next release. Best regards, Marek From: Ruslan Velkov rus...@sirma.bg To: Marek Šurek marek_su...@yahoo.co.uk Cc: owlim-discussion@ontotext.com owlim-discussion@ontotext.com; Barry Bishop barry.bis...@ontotext.com Sent: Tuesday, 19 June 2012, 13:51 Subject: Re: [Owlim-discussion] Poor performance for Sparql queries with property path optional elements Hi Marek, The new behaviour after we introduced the QueryJoinOptimizer is a feature we were looking for and it took much time to be implemented and stabilized in Sesame because in the beginning it was implemented like a different query model node (SPARQLIntersection), then this node disappeared from the Sesame code base and the implementation was introduced in EvaluationStrategyImpl.evaluate(Join), see http://www.openrdf.org/issues/browse/SES-953. This issue was fixed in Sesame 2.6.5 and the fix was available for Owlim 5.0, but our release notes were rather large because they covered all bug fixes and new features, so we didn't mention the concrete Sesame bug fixes and improvements, just mentioned that we use Sesame version 2.6.5 (that is why you can't find this issue in our release notes). Some of the sub-select handling code went into the QueryJoinOptimizer, but this happened after releasing Owlim 5.0 and was available in Sesame 2.6.6, from where we took that optimizer and introduced it into our code (version 5.1). The thing which was really improved was namely the sub-selects, we experimented with the BSBM Business Intelligence benchmark in particular, and that optimizer gave wonderful results (some of the queries contained sub-selects, sub-select nested into sub-selects, sub-selects interconnected just with a single filter, etc.). I hope this answers your question what actually was improved by using this optimizer. The bad think with this optimizer is that it touches the whole query model, not just the sub-selects, hence the side effects we are experiencing. After communicating the issue with Jeen it became clear that apart from several small fixes in Sesame, we may be able to fix this issue in Owlim by supplying statistics to the QueryJoinOptimizer in order to influence its decision-making when reordering the query. However, the statistics that we use in our query optimization strategy are much different from what Sesame uses in its own one, so it may take some time. If it is achievable, you'll be notified with the fix. Cheers, Ruslan On 06/18/2012 07:18 PM, Marek Šurek wrote: Hi Ruslan, we have the similar issue (see http://www.mail-archive.com/owlim-discussion@ontotext.com/msg01626.html), which is probably based on the same thing. I don't understand from your response whether the new behaviour is bug or feature. I carefully looked at release notes and nothing so serious as total change query optimizer, which dramatically changes behaviour of subselects, was not mentioned! I see(and really appreciate) that you are looking for shorttime hotfix solution introducing some parameter but for now I don't know what to do. Lot of our bussiness code uses subselects as it is powerful feature. We rely on your answer whether this feature/bug is just problem of few days or it is permanent state. The parameter is great as hotfix, but I can't test and optimize all queries and try in which case it runs faster and after some minor bugfix suddenly without any warning I can start from begining. As it is not well documented, I would like to ask, what are positive aspects of using new query optimizer? Where can we see improvements? What kind of queries should ran faster? Thank you for your time and looking forward for your response. Best regards, Marek From: Ruslan Velkov rus...@sirma.bg To: owlim-discussion@ontotext.com Sent: Monday, 18 June 2012, 15:53 Subject: Re: [Owlim-discussion] Poor performance for Sparql queries with property path optional elements Hi Krzysztof, Many thanks for reporting this and for providing a test class! What we introduced in 5.1 is the Sesame's QueryJoinOptimizer which rearranges joins so that if there are sub-select clauses in the query they will be evaluated first and in the best possible order
Re: [Owlim-discussion] Poor performance for Sparql queries with property path optional elements
Hi Ruslan, we have the similar issue (see http://www.mail-archive.com/owlim-discussion@ontotext.com/msg01626.html), which is probably based on the same thing. I don't understand from your response whether the new behaviour is bug or feature. I carefully looked at release notes and nothing so serious as total change query optimizer, which dramatically changes behaviour of subselects, was not mentioned! I see(and really appreciate) that you are looking for shorttime hotfix solution introducing some parameter but for now I don't know what to do. Lot of our bussiness code uses subselects as it is powerful feature. We rely on your answer whether this feature/bug is just problem of few days or it is permanent state. The parameter is great as hotfix, but I can't test and optimize all queries and try in which case it runs faster and after some minor bugfix suddenly without any warning I can start from begining. As it is not well documented, I would like to ask, what are positive aspects of using new query optimizer? Where can we see improvements? What kind of queries should ran faster? Thank you for your time and looking forward for your response. Best regards, Marek From: Ruslan Velkov rus...@sirma.bg To: owlim-discussion@ontotext.com Sent: Monday, 18 June 2012, 15:53 Subject: Re: [Owlim-discussion] Poor performance for Sparql queries with property path optional elements Hi Krzysztof, Many thanks for reporting this and for providing a test class! What we introduced in 5.1 is the Sesame's QueryJoinOptimizer which rearranges joins so that if there are sub-select clauses in the query they will be evaluated first and in the best possible order (in terms of number of variables shared by their respective projections). This optimizer allows for fast and efficient evaluation of nested SELECT clauses. What I saw as a by-product of this optimizer on q2 was this: [Original query plan without applying the QueryJoinOptimizer] Projection ProjectionElemList ProjectionElem name Join Join StatementPattern Var (name=-const-1, value=person://1, anonymous) Var (name=-const-2, value=http://xmlns.com/foaf/0.1/knows, anonymous) Var (name=-const-2-0, anonymous) Union ZeroLengthPath Var (name=-const-2-0, anonymous) Var (name=-const-3-1, anonymous) StatementPattern Var (name=-const-2-0, anonymous) Var (name=-const-3, value=http://xmlns.com/foaf/0.1/knows, anonymous) Var (name=-const-3-1, anonymous) StatementPattern Var (name=-const-3-1, anonymous) Var (name=-const-4, value=http://xmlns.com/foaf/0.1/name, anonymous) Var (name=name) [Query plan after applying the QueryJoinOptimizer] Projection ProjectionElemList ProjectionElem name Join StatementPattern Var (name=-const-1, value=person://1, anonymous) Var (name=-const-2, value=http://xmlns.com/foaf/0.1/knows, anonymous) Var (name=-const-2-0, anonymous) Join StatementPattern Var (name=-const-3-1, anonymous) Var (name=-const-4, value=http://xmlns.com/foaf/0.1/name, anonymous) Var (name=name) Union ZeroLengthPath Var (name=-const-2-0, anonymous) Var (name=-const-3-1, anonymous) StatementPattern Var (name=-const-2-0, anonymous) Var (name=-const-3, value=http://xmlns.com/foaf/0.1/knows, anonymous) Var (name=-const-3-1, anonymous) As you can see, using the first query model we'll evaluate person://1 http://xmlns.com/foaf/0.1/knows -const-2-0 and then -const-2-0 http://xmlns.com/foaf/0.1/knows -const-3-1 with the already bound -const-2-0 and finally we'll evaluate the last statement using the binding for -const-3-1. This is the correct ordering. What we can see from the second query model is evaluating person://1 http://xmlns.com/foaf/0.1/knows -const-2-0 and then the last statement -const-3-1 http://xmlns.com/foaf/0.1/name name and then the optional second statement, but as far as the the first two patterns in this order don't share any variables a Cartesian product will be formed (the first pattern has 999 results and the second one has 23334 results, hence 23,310,666 iterations, very few of which will succeed). Unfortunately, there is no way to turn this optimizer off and even there was one, other queries would become much slower (namely the ones with sub-selects). There can be introduced a parameter that switches the optimizer off as a workaround, but you may encounter problems when using queries with sub-selects, so in that case you should arrange the sub-selects manually and they should be the first thing to a appear in a query (in case you
[Owlim-discussion] Query execution problem between b5123-b5183
Hi, I just recieved the lasted build Owlim 5.1 b5183 and deploy it on our server to test it. I found strange bug related to query execution. The query(INSERT query) took 9 seconds with OWLIM 5.0 b5123(with Sesame 2.6.6), but with build b5183 it runs more that 30 minutes and it is still not at the end. Also memory consumption is enormous. I realized that between these two builds you fixed bug when removeRepository, introdruced in Sesame 2.6.6, was not working ( Internal server error 500 ). I hope that at least I minimized the set of possible places where bug could be made, as I don't know if I can reproduce you the scenario because of private character of the data. Best regards, Marek___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
Re: [Owlim-discussion] Problem with OWLIM, Sesame, Tomcat
Hi, if you don't need to do it through workbench I suggest far easier way to create custom ruleset. My solution is connected to windows... windows 7 64bit 1. Deploytwo official war-files included in owlim distribution .zip file (in folder /sesame_owlim ) by copying them into webapps directory in Tomcat (make sure you delete your previous attempt described in your mail) 2. Unpack console.zip (also in folder /sesame_owlim in zip file) 3. Run console through console.bat(or .sh depends on your OS) (.../openrdf-sesame-console/bin/console.bat) 4. Your first command in console should be : connect http://localhost:8080/openrdf-sesame. 5. Then try whether other repositories already exist by command : show r. a) it there exists repository with the same name you want to create run command : drop nameOfYourRepository. 6. Copy your .ttl file into C:/Users/YourAccountName/AppData/Roaming/Aduna/OpenRDF sesame console/templates/ (notice : folder /AppData is in windows normally hidden) 7. Open console and run command (dont forget to run step 4 - connect to localhost) create nameOfYourRepository. Notice : nameOfYourRepository should be equal to name of your .ttl file which you should copy into .../templates folder (see step 6) Notice : every command in console should end with dot (.) After these steps you will see your repository also in workbench but it will have your custom ruleset specified in your .ttl file and no error should occur. Workbench will normally interact with such repository, so dont worry about it :). Hope it helps, Marek From: Babis Doulaverakis doula...@yahoo.gr To: Ontotext owlim-discussion@ontotext.com Sent: Thursday, 5 April 2012, 12:50 Subject: [Owlim-discussion] Problem with OWLIM, Sesame, Tomcat Hello, I am having a problem when I try to create an OWLIM repository using openrdf-workbench. The repository looks like it is created (i.e. it appears in the list for repositories in workbench) but when i try to access Namespaces for example I get the following error from Tomcat: java.lang.IllegalStateException: Connection factory has been shutdown. org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(MultiThreadedHttpConnectionManager.java:463) org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(MultiThreadedHttpConnectionManager.java:416) org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:153) org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) org.openrdf.http.client.HTTPClient.getTupleQueryResult(HTTPClient.java:1099) org.openrdf.http.client.HTTPClient.getNamespaces(HTTPClient.java:906) org.openrdf.http.client.HTTPClient.getNamespaces(HTTPClient.java:887) org.openrdf.repository.http.HTTPRepositoryConnection.getNamespaces(HTTPRepositoryConnection.java:420) org.openrdf.workbench.commands.NamespacesServlet.service(NamespacesServlet.java:48) org.openrdf.workbench.base.TransformationServlet.service(TransformationServlet.java:94) org.openrdf.workbench.base.TransformationServlet.service(TransformationServlet.java:73) org.openrdf.workbench.base.BaseServlet.service(BaseServlet.java:40) org.openrdf.workbench.proxy.ProxyRepositoryServlet.service(ProxyRepositoryServlet.java:93) org.openrdf.workbench.proxy.WorkbenchServlet.service(WorkbenchServlet.java:131) org.openrdf.workbench.proxy.WorkbenchServlet.service(WorkbenchServlet.java:90) org.openrdf.workbench.proxy.WorkbenchGateway.service(WorkbenchGateway.java:109) org.openrdf.workbench.base.BaseServlet.service(BaseServlet.java:40) org.openrdf.workbench.proxy.CookieCacheControlFilter.doFilter(CookieCacheControlFilter.java:52) Could you please help me on this? What I have done is: - Copied owlim-lite-4.3.jar in WEB-INF\lib folder of both openrdf-workbench and openrdf-sesame - Copied owlim.ttl to openrdf-workbench\WEB-INF\lib\org\openrdf\console - Copied create-owlim.xsl to openrdf-workbench\transformations - Accessed http://localhost:8080/openrdf-workbench/repositories/NONE/create?type=owlim in order to create the repository The owlim.ttl and create-owlim.xsl were copied from http://www.mail-archive.com/owlim-discussion@ontotext.com/msg00411.html I am using Tomcat 6.0.29, Java JDK 1.6.0.29, sesame 2.6.3, owlimlite 4.3 What I eventually want to do is create an owlim repository using a custom rule set. Thank you, Babis ___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
Re: [Owlim-discussion] Problem with OWLIM, Sesame, Tomcat
Hi, I guess it is stored in builtin_Rules.pie as commentary in the file reference to file as Max. There is division commentary line in builtin_Rules.pie which divide rules into Horst and Max. See : ... ... //== // This rule file contains both OWL-Horst rules and OWL-Max rules. // The OWL-Horst rule set is a subset of the OWL-Max rules // The rules above this point all belong to OWL-Horst AND OWL-Max // The rules below this point only belong to OWL-Max // The next line is the dividing point: // OWL-Max supporting rules //== ... ... I'm not expert in the field of how Owlim stores rules, but what can I really say 9200 rules is too many rules for one repository. TRREE is java based rule engine and during compilation it probably creates highly optimized java structures which could take such big amount of memory and java garbage collector can't trash them as they had to be used everytime new triples are inserted into repository...but it is only my guess. We can only hope TRREE (owlim's reasoner) can procede so many rules in acceptable amount of time (thought it is extremely fast rule engine and with owlim database and sesame it is probably the fastest solution on the market). I can share my experience with real application + Lubm test. My notebook config: P8600 Core 2 Duo, 2,4Ghz 4GB RAM Win 7 64bit Tomcat 7 Lubm 50 test (50 universities - around 5M explicit triples) with horst-optimized loaded on my 832sec with Horst ruleset, with OWL2 RL 3200sec and it contains only less than 100 rules. Complexity of rule evaluation rises dramaticly as number of new rules are added. In real applicatoin loading + inferencing of 4M explicit triples with only 4 rules which create total number of 6.5M triples (2,5M implicit triples) takes on my notebook 802sec. Of course complexity of rules is also very important, I just want to say that 9200 could be many. Number of rules indicates you need it for some research, in this case ontotext provides Owlim SE trial evaluation copy(limited to short amount of time) which is more optimized in... everything including TRREE :). You just need to apply and provide some details of your reaseach. Marek From: Babis Doulaverakis doula...@yahoo.gr To: Marek Šurek marek_su...@yahoo.co.uk; Ontotext owlim-discussion@ontotext.com Sent: Thursday, 5 April 2012, 14:26 Subject: Re: [Owlim-discussion] Problem with OWLIM, Sesame, Tomcat Thanks Marek, I managed to create the OWLIM repository with a custom rule set. I am curious about one thing though. The rule set I am using contains around 9200 rules. I appended those rules in the Builtin_rules.pie file (in order to have OWL reasoning) so I now have a huge rule base. OWLIM successfully loaded the rule base but the memory consumption is huge even with no OWL file loaded. Tomcat consumes 940MB of memory (normally it is around 150MB). The memory consumption rises fast when the rule base is compiled but does not drop after compilation has finished. Is this normal behavior? I restarted Tomcat and when I try to access the repository I created, the rules are compiled again which again leads to huge memory consumption. On more thing, in which .pie file that comes with OWLIM-lite 4.3 distribution are OWL/Max rules stored? Thanks, Babis From: Marek Šurek marek_su...@yahoo.co.uk To: Babis Doulaverakis doula...@yahoo.gr; Ontotext owlim-discussion@ontotext.com Sent: Thursday, April 5, 2012 2:27 PM Subject: Re: [Owlim-discussion] Problem with OWLIM, Sesame, Tomcat Hi, if you don't need to do it through workbench I suggest far easier way to create custom ruleset. My solution is connected to windows... windows 7 64bit 1. Deploytwo official war-files included in owlim distribution .zip file (in folder /sesame_owlim ) by copying them into webapps directory in Tomcat (make sure you delete your previous attempt described in your mail) 2. Unpack console.zip (also in folder /sesame_owlim in zip file) 3. Run console through console.bat(or .sh depends on your OS) (.../openrdf-sesame-console/bin/console.bat) 4. Your first command in console should be : connect http://localhost:8080/openrdf-sesame. 5. Then try whether other repositories already exist by command : show r. a) it there exists repository with the same name you want to create run command : drop nameOfYourRepository. 6. Copy your .ttl file into C:/Users/YourAccountName/AppData/Roaming/Aduna/OpenRDF sesame console/templates/ (notice : folder /AppData is in windows normally hidden) 7. Open console and run command (dont forget to run step 4 - connect to localhost) create nameOfYourRepository. Notice : nameOfYourRepository should be equal to name of your .ttl file which you should copy into .../templates folder (see step 6
Re: [Owlim-discussion] CRITICAL FAILURE !!! The database restorer tool has already failed once before to perform image restore.
Hi Monika, I also experienced this exception number of times on OWLIM 4.3. The problem appears only when I ran openrdf-sesame and client application at the same Tomcat server and due to developing client application I had to restart Tomcat very offen to see changes in my client application. My solution was to divide those two applications into different tomcat instances so now they run independently(my client application runs on different Tomcat as openrdf-sesame) and I don't restart Tomcat instance with openrdf-sesame.war almost never. I hope this would help, Marek From: Monika Solanki msolanki.maili...@gmail.com To: owlim-discussion@ontotext.com Sent: Saturday, 31 March 2012, 10:59 Subject: [Owlim-discussion] CRITICAL FAILURE !!! The database restorer tool has already failed once before to perform image restore. Hello Again, A couple of times I have had this error thrown to me and I have had to populate the repository again. CRITICAL FAILURE !!! The database restorer tool has already failed once before to perform image restore. I would appreciate any help on how I can possibly avoid this error. Thanks, Monika ___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
Re: [Owlim-discussion] Consistency checks with corollaries not implemented
Hi, I think the problem is with your understanding what you should expect from consistency checks. Consistency checks are here to reveal inconsistency in repository/ontology. Therefore they don't produce any consequence (corollary) they just alert in case of inconsistency. If you want to create consequence you should create rule. In you case it should look like this : Id: yourRangeRule x p y p rdfs:range c - y rdf:type c Marek From: Krzysztof Sielski siel...@man.poznan.pl To: Owlim discussion owlim-discussion@ontotext.com Sent: Wednesday, 29 February 2012, 16:29 Subject: [Owlim-discussion] Consistency checks with corollaries not implemented Hello, I added a consistency check to the rule definition file as specified in the user guide: Consistency: range x p y p rdfs:range c - y rdf:type c Unfortunatelly, when the .pie file is compiled, an Exception is thrown: java.lang.RuntimeException: Consistency checks with corollaries not implemented I am using Owlim SE 4.3.4423. Is it a functionality that will be implemented in future releases? -- Regards, Krzysztof Sielski Poznan Supercomputing and Networking Center ___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion___ Owlim-discussion mailing list Owlim-discussion@ontotext.com http://ontomail.semdata.org/cgi-bin/mailman/listinfo/owlim-discussion
Re: [Owlim-discussion] Problem with jdk1.7.0_02
Hi, from my experience there is problem with jdk1.7.x and Sesame. In my case I can't simply run Sesame Console, so I think the issue is connected somehow with sesame. Try to replace your version of sesame lib, with the new one 2.6.3(it is not mentioned on main page but it is in download repository). Maybe your problem will dissapear. Marek___ OWLIM-discussion mailing list OWLIM-discussion@ontotext.com http://ontotext.com/mailman/listinfo/owlim-discussion
