Re: [Virtuoso-users] Memory leaks with virtuoso 7.2.0

[Davis, Daniel (NIH/NLM) [C]]

Hugh Williams wrote:
 Why were the 7.2.1  7.2.0 builds configured with different options, as they 
 not both being performed on the same OS, in which case I would expect the 
 configure options to the same ?

I tried the same configure options as well, and then I tried different options 
because I was grasping at straws.   Here is a graph showing that 6.1.8 does not 
have the problem, that the changes to configure options doesn't matter for this 
issue.

[cid:image003.png@01D0D8D6.3F850EB0]

The basics for reproducing the problem are as follows:

*Download ftp://ftp.nlm.nih.gov/online/mesh/mesh.nt and the vocabulary 
and void files from that location.

*Load the vocabulary-0.9.ttl to graph http://id.nlm.nih.gov/mesh/vocab

*Load the void-0.9.ttl to graph http://id.nlm.nih.gov/mesh/void

*Load the mesh.nt file to graph http://id.nlm.nih.gov/mesh again and 
again (I used 15 seconds delay) with DB.DBA.ttlp().

From: Hugh Williams [mailto:hwilli...@openlinksw.com]
Sent: Sunday, August 16, 2015 8:52 PM
To: Davis, Daniel (NIH/NLM) [C] daniel.da...@nih.gov
Cc: virtuoso-users@lists.sourceforge.net
Subject: Re: [Virtuoso-users] Memory leaks with virtuoso 7.2.0

Hi Daniel,

Why were the 7.2.1  7.2.0 builds configured with different options, as they 
not both being performed on the same OS, in which case I would expect the 
configure options to the same ?

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers

On 13 Aug 2015, at 20:39, Davis, Daniel (NIH/NLM) [C] 
daniel.da...@nih.govmailto:daniel.da...@nih.gov wrote:

Hugh Williams wrote:

Also, note the following 2 parameters: ThreadCleanupInterval, 
ResourcesCleanupInterval which can both be set to 1 in order to reduce memory 
leaking:


http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ex_threadcleanupinterval

although as this  was not apparently occurring with 7.2.0 it may not help, but 
worth trying ...

It does occur with 7.2.0 - I'd vainly hoped 7.2.1 would fix it.The XSLT 
process produces the file ftp://ftp.nlm.nih.gov/online/mesh/mesh.nt, which 
contains 11,917,672 triples.
I'm also using the ontology 
ftp://ftp.nlm.nih.gov/online/mesh/vocabulary_0.9.ttl and the void file 
ftp://ftp.nlm.nih.gov/online/mesh/void_0.9.ttl.

I cannot actually say it is a memory leak because I haven't pulled out those 
tools yet - VmRSS seems to grow linearly over time.
VmSize does the same, only the sloop is not as steep.

I've built Virtuoso 7.2.1 on CentOS 6.6 as follows:
./configure --prefix=whatever --with-readline

Previously, with 7.2.0, my build was:
export CFLAGS=-O2 -m64
./configure --prefix=whatever --with-readline --with-iodbc=/usr 
--without-internal-zlib --enable-shared --disable-static

I've also configured it with ThreadCleanupInterval = 1 and 
ResourcesCleanupInterval = 1.

The reload I'm testing runs the following commands from an isql command-line:

log_enable(2,1);
sparql clear graph http://id.nlm.nih.gov/mesh;
sparql clear graph http://id.nlm.nih.gov/mesh/vocab;
sparql clear graph http://id.nlm.nih.gov/mesh/void;
DB.DBA.ttlp(file_to_string_output('path-to-files/mesh.nt'), '', 
'http://id.nlm.nih.gov/mesh', 0);

DB.DBA.ttlp(file_to_string_output('path-to-files/vocabulary_0.9.ttl'), '', 
'http://id.nlm.nih.gov/mesh/vocab', 0);
rdfs_rule_set('http://id.nlm.nih.gov/mesh/vocab', 
'http://id.nlm.nih.gov/mesh/vocab');
DB.DBA.ttlp(file_to_string_output('path-to-files/void_0.9.ttl'), 
'', 'http://id.nlm.nih.gov/mesh/void', 0);
CHECKPOINT;

I'm retrying with a shorter script, which only loads 100,000 triples:

log_enable(2,1);
sparql clear graph http://id.nlm.nih.gov/meshtest;
DB.DBA.ttlp(file_to_string_output('path-to-files/small-mesh.nt', 
'', 'http://id.nlm.nih.gov/meshtest', 0);

If that proves stable, I'll do the following:
- stop clearing and reloading the vocabulary and void graphs and redefining the 
ruleset
- let you guys know it worked.

If that is not stable, I'll do one or several of the following:
- try virtuoso-opensource-6
- open/comment on an issue on 
github.com/openlinksw/virtuoso-opensource/issueshttp://github.com/openlinksw/virtuoso-opensource/issues
- install valgrind and valgrind-devel and build with malloc debug
- evaluate switching to the directory scan option.

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

[Virtuoso-users] Birthday problem - RDF_64BIT_UPGRADE()

[Davis, Daniel (NIH/NLM) [C]]

http://docs.openlinksw.com/virtuoso/rdfdatarepresentation.html states that:


Virtuoso supports a native IRI_ID data type, internally an unsigned 32 bit or 
unsigned 64 bit integer value. Small databases can use 32 bit values but if 
database becomes big then the administrator should 
executeDB.DBA.RDF_64BIT_UPGRADE() procedure that will switch to 64-bit values.

I've loaded 2 million subjects - accordingly it makes sense that I'd need 
64-bit hashes.When I try to call DB.DBA.RDF_64BIT_UPGRADE(), I get an error 
that it is not defined.  It occurs to me that:

1. This may be a feature of Virtuoso Professional

2. OpenLink Sw. may be using 64-bit hashes all the time (more sensible in 
my view).
And then there may also be a problem with my installation, e.g. some configure 
option for same.

I haven't observed any collisions, I'm just digging in and have a concern.

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Memory leaks with virtuoso 7.2.0

[Davis, Daniel (NIH/NLM) [C]]

Hugh Williams wrote: 
 Also, note the following 2 parameters: ThreadCleanupInterval, 
 ResourcesCleanupInterval which can both be set to 1 in order to reduce memory 
 leaking:
 
   
 http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ex_threadcleanupinterval
 
 although as this  was not apparently occurring with 7.2.0 it may not help, 
 but worth trying ...

It does occur with 7.2.0 - I'd vainly hoped 7.2.1 would fix it.The XSLT 
process produces the file ftp://ftp.nlm.nih.gov/online/mesh/mesh.nt, which 
contains 11,917,672 triples.
I'm also using the ontology 
ftp://ftp.nlm.nih.gov/online/mesh/vocabulary_0.9.ttl and the void file 
ftp://ftp.nlm.nih.gov/online/mesh/void_0.9.ttl.

I cannot actually say it is a memory leak because I haven't pulled out those 
tools yet - VmRSS seems to grow linearly over time.
VmSize does the same, only the sloop is not as steep.

I've built Virtuoso 7.2.1 on CentOS 6.6 as follows:
./configure --prefix=whatever --with-readline

Previously, with 7.2.0, my build was:
export CFLAGS=-O2 -m64
./configure --prefix=whatever --with-readline --with-iodbc=/usr 
--without-internal-zlib --enable-shared --disable-static

I've also configured it with ThreadCleanupInterval = 1 and 
ResourcesCleanupInterval = 1. 

The reload I'm testing runs the following commands from an isql command-line:

log_enable(2,1);
sparql clear graph http://id.nlm.nih.gov/mesh;
sparql clear graph http://id.nlm.nih.gov/mesh/vocab;
sparql clear graph http://id.nlm.nih.gov/mesh/void;
DB.DBA.ttlp(file_to_string_output('path-to-files/mesh.nt'), '', 
'http://id.nlm.nih.gov/mesh', 0);
DB.DBA.ttlp(file_to_string_output('path-to-files/vocabulary_0.9.ttl'), 
'', 'http://id.nlm.nih.gov/mesh/vocab', 0);
rdfs_rule_set('http://id.nlm.nih.gov/mesh/vocab', 
'http://id.nlm.nih.gov/mesh/vocab');
DB.DBA.ttlp(file_to_string_output('path-to-files/void_0.9.ttl'), '', 
'http://id.nlm.nih.gov/mesh/void', 0);
CHECKPOINT;

I'm retrying with a shorter script, which only loads 100,000 triples:

log_enable(2,1);
sparql clear graph http://id.nlm.nih.gov/meshtest;
DB.DBA.ttlp(file_to_string_output('path-to-files/small-mesh.nt', '', 
'http://id.nlm.nih.gov/meshtest', 0);

If that proves stable, I'll do the following:
 - stop clearing and reloading the vocabulary and void graphs and redefining 
the ruleset
 - let you guys know it worked.

If that is not stable, I'll do one or several of the following:
 - try virtuoso-opensource-6
 - open/comment on an issue on github.com/openlinksw/virtuoso-opensource/issues
 - install valgrind and valgrind-devel and build with malloc debug
 - evaluate switching to the directory scan option.

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH




--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Memory leaks with virtuoso 7.2.0

[Davis, Daniel (NIH/NLM) [C]]

Hugh Williams wrote:
 How is this process to convert MeSH XML to MeSH RDF which uses lots of 
 memory performed, is this done in Virtuoso or by some external process ?

It's really converting to triples - https://github.com/HHS/meshrdf contains the 
conversion code, which is basically an XSLT process run by saxon9he in java.  
There is no Java code in our converter.One reason we use the current method 
is so that researchers and linked data software professionals can independently 
do the conversion.

Could this XSLT code run inside of Virtuoso?   How would we do a partial change 
every night rather than a full reload given that some IRIs that function as 
both subjects and objects may be deleted each night? 

Thanks,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH



--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] log_enable(2,1) - how long?

[Davis, Daniel (NIH/NLM) [C]]

Thanks, it obviously had to be scoped, but it wasn't clear to me how it was 
scoped.If I call log_enable(2,1) manually before the stored procedure, 
would the entire stored procedure turn off transaction logging?

If I refactor my stored procedure to use fewer statements to do the same 
changes to the graph, will that cause the transaction logging to be less of an 
issue for performance (and failure as it exceeds its maximum size)?

From: Hugh Williams [mailto:hwilli...@openlinksw.com]
Sent: Tuesday, October 27, 2015 11:20 PM
To: Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov>
Cc: virtuoso-users <virtuoso-users@lists.sourceforge.net>
Subject: Re: [Virtuoso-users] log_enable(2,1) - how long?

Hi Daniel,

log_enable (2,1) would turn off transaction logging for the next statement 
executed which on completion the transaction logging would be turned back on.

If it is called in a stored procedure then once again it would turn off 
transaction logging for the next statement executed which on completion the 
transaction logging would be turned back on.

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers

On 27 Oct 2015, at 13:53, Davis, Daniel (NIH/NLM) [C] 
<daniel.da...@nih.gov<mailto:daniel.da...@nih.gov>> wrote:

When I execute log_enable(2,1) in a session, does it just affect statements 
made by that statement, or the whole server?
When the session closes, does it revert to logging transactions?

What if I move log_enable(2,1) to a stored procedure called by a session?

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net<mailto:Virtuoso-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] How can I profile stored procedures?

[Davis, Daniel (NIH/NLM) [C]]

How can I profile SQL or SPARQL statements when running this statements within 
a stored procedure?

I've refactored my versioning algorithm to use more complicated single queries, 
for example:

LOG_ENABLE(2, 1);
DELETE FROM RDF_QUAD
  WHERE g = __target_graph
AND EXISTS (SELECT subject_iid
  FROM msh_update
 WHERE msh_update.graph_iid = RDF_QUAD.g
   AND msh_update.subject_iid = RDF_QUAD.s
   AND status = 1);

 LOG_ENABLE(2,1);
INSERT INTO RDF_QUAD (g, s, p, o)
SELECT u.graph_iid, u.subject_iid, q.p, q.o
  FROM msh_update u
  JOIN rdf_quad q ON u.subject_iid = q.s
 WHERE u.graph_iid = __target_graph
   AND q.g = __update_graph AND status IN ('upate', 'insert');

Thanks,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] log_enable(2,1) - how long?

[Davis, Daniel (NIH/NLM) [C]]

When I execute log_enable(2,1) in a session, does it just affect statements 
made by that statement, or the whole server?
When the session closes, does it revert to logging transactions?

What if I move log_enable(2,1) to a stored procedure called by a session?

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] federated SPARQL queries?

[Davis, Daniel (NIH/NLM) [C]]

Bassam,

As a professional in this field, I want to suggest that SPARQL Federated Query 
is not widely used in the field.   Think about how this affects inferencing - 
unless the federating server has some way of knowing which vocabulary to 
associate with a remote graph, there is no way to do inferencing.   Think also 
about Distributed Systems Theory in general – how should your SPARQL endpoint 
behave when another SPARQL endpoint is not responding or is slow?   From a 
distributed systems theory perspective, stopping failures such as “not 
responding” are much, much more tractable than non-conformant/random behavior.

There is an analogous problem in distributed Information Retrieval (IR), e.g. 
text search.It is common for cooperating search servers to communicate Term 
Frequency (TF) and Document Frequency(DF) on each search server among cluster 
members so that distributed relevancy may be calculated (usually described as 
TF-IDF).With true Federated Search/Meta-search, this isn’t possible, and so 
search relevancy suffers.It has given federated search rather a black eye 
although there are many scholarly articles about it.

In the semantic web, the common solution is to offer both a download of TTL, 
RDF, or NTriples and have other SPARQL endpoints include your downloads, as 
Hugh Williams suggests.Although my semantic web site, 
http://id.nlm.nih.gov/mesh is a beta and likely to remain that way, we offer 
downloads of NTriples files at ftp://ftp.nlm.nih.gov/online/mesh.

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH



From: Hugh Williams [mailto:hwilli...@openlinksw.com]
Sent: Monday, November 16, 2015 8:59 AM
To: Eng.Bassam 
Cc: virtuoso-users@lists.sourceforge.net
Subject: Re: [Virtuoso-users] federated SPARQL queries?

Hi Bassam,

I assume your ontology is already in RDF format, in which case see the 
following article on how to import/load and Ontology into Virtuoso:


http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtTipsAndTricksGuideImportOntology

or generally you can use any of the many methods of insert/load/import’ing RDF 
data into Virtuoso detailed at:


http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFInsert

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers

On 16 Nov 2015, at 07:48, Eng.Bassam 
> wrote:

Hi
I'm Bassam a software engineer, I'm preparing master thesis in semantic web

I have my medical data in virtuoso and want to link my data to external 
ontology for diseases,
how can I do that :

must upload the ontology in virtuoso server, if yes how do that?

my objective is make federated SPARQL queries to traversal on my data also 
diseases ontology

thanks.

--
---
> Eng.Bassam Najeeb.
>Software Engineer.
--
Presto, an open source distributed SQL query engine for big data, initially
developed by Facebook, enables you to easily query your data on Hadoop in a
more interactive manner. Teradata is also now providing full enterprise
support for Presto. Download a free open source copy now.
http://pubads.g.doubleclick.net/gampad/clk?id=250295911=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

--
Presto, an open source distributed SQL query engine for big data, initially
developed by Facebook, enables you to easily query your data on Hadoop in a 
more interactive manner. Teradata is also now providing full enterprise
support for Presto. Download a free open source copy now.
http://pubads.g.doubleclick.net/gampad/clk?id=250295911=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] output:format "NT" question

[Davis, Daniel (NIH/NLM) [C]]

I have a simple script to export triples, but it seems not to get all triples, 
but only about 10,000.I cannot guess what may be the problem.
Any advice is appreciated.

isql  dba "$PASSWORD" BANNER=OFF BLOBS=ON VERBOSE=OFF ECHO=OFF PROMPT=OFF 
TIMEOUT=0 >fullmesh.nt 
CONSTRUCT { ?s ?p ?o }
WHERE {
  GRAPH  {
?s ?p ?o
  }
};
EOF


Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
Presto, an open source distributed SQL query engine for big data, initially
developed by Facebook, enables you to easily query your data on Hadoop in a 
more interactive manner. Teradata is also now providing full enterprise
support for Presto. Download a free open source copy now.
http://pubads.g.doubleclick.net/gampad/clk?id=250295911=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] Validate graph based on rules

[Davis, Daniel (NIH/NLM) [C]]

So, I have a rule set (vocabulary) for a graph that includes certain object 
properties, some with a domain and range, and some with only domain.   I also 
have object properties.How can a go forward and validate such a schema 
using Virtuoso?Can I declare the graph as not validating, and then validate 
only after load, (e.g. making validation of data conversion a test that is done 
only in development and QA)?

Some simple examples would be - dc:SimpleLiteral may not have a language class. 
  xsd:date must have an understandable format.xsd:boolean must only have 
"true" and "false", 1 and 0.

Thanks,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] power user problems

[Davis, Daniel (NIH/NLM) [C]]

I guess I am turning into more of a power-user.   Nonetheless, I have had some 
problems using exec() with SPARQL statements where the GRAPH URI is a variable. 
  In this case, I must use { GRAPH ?? { ?? mesh:altLabel ?? } } for instance as 
below.   I also see no way to specify the language code when I am updating 
strings in this way except for sprintf() and then SPARQL_EVAL - I do not think 
old_label carries a language.   Advice is appreciated, but not expected.

Note that I do not really need the loop for rdfs:label, but I will need it 
later for a custom subProperty of rdfs:label which is meshv:altLabel which does 
have cardinality >1.

CREATE PROCEDURE meshrdf_prefix_label(in graph_uri varchar, in subj_uri 
varchar, in prefix varchar) {
DECLARE state, msg, descs, chandle, arow any;

state := '0';
exec('SPARQL PREFIX rdfs: 
  SELECT ?label WHERE { GRAPH ?? { ?? rdfs:label ?label . 
FILTER(lang(?label) = \'en\') } }',
 state, msg, vector(graph_uri, subj_uri), 100, descs, null, chandle);

IF (state <> '0')
signal(state, msg);

WHILE (exec_next(chandle, state, msg, arow) = 0) {
IF (state <> '0')
signal(state, msg);

DECLARE old_label, new_label varchar;
old_label := arow[0];
new_label := sprintf('%S %S', prefix, old_label);

DECLARE stmt varchar;

stmt := sprintf('SPARQL PREFIX rdfs: 

 WITH <%S> DELETE { <%S> rdfs:label "%S"@en } INSERT { 
<%S> rdfs:label "%S"@en }',
graph_uri, subj_uri, old_label, subj_uri, new_label);

exec(stmt, state, msg, vector(), 1, null, null);
IF (state <> '0')
signal(state, msg);

}
exec_close(chandle);
}

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] federated SPARQL queries?

[Davis, Daniel (NIH/NLM) [C]]

Bassam,

I think I came on a little strong.   While I’m an IT practitioner in this 
field, I’m not a researcher.   You are doing research. NLM’s MeSH RDF 
Pilot, like other federal IT projects, has to be careful about security and 
PII.   Lacking the budget and time to vet different SPARQL endpoints, we of 
course turn federation off.In general, anyone using federation must think 
about the quality of the modeling, data, and implementation of a SPARQL 
endpoint that may be included, and any implicit contract with their users. 
However, these concerns can be addressed in other ways than simply importing 
the data.

I still have an opinion on how SPARQL as a protocol could be strengthened – I 
am very impressed with the language and RDF modeling, but as a protocol I wish 
there were stronger error responses and higher level ways to run SPARQL 
queries, such as simple POJO bindings to queries, and better ways to page 
through a set of results.These are middleware issues that can be addressed. 
   JDBC, for example, came long after SQL was well established.

Hoping this helps,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

From: Davis, Daniel (NIH/NLM) [C]
Sent: Monday, November 16, 2015 11:04 AM
To: 'Hugh Williams' <hwilli...@openlinksw.com>; Eng.Bassam <bassa...@gmail.com>
Cc: virtuoso-users@lists.sourceforge.net
Subject: RE: [Virtuoso-users] federated SPARQL queries?

Bassam,

As a professional in this field, I want to suggest that SPARQL Federated Query 
is not widely used in the field.   Think about how this affects inferencing - 
unless the federating server has some way of knowing which vocabulary to 
associate with a remote graph, there is no way to do inferencing.   Think also 
about Distributed Systems Theory in general – how should your SPARQL endpoint 
behave when another SPARQL endpoint is not responding or is slow?   From a 
distributed systems theory perspective, stopping failures such as “not 
responding” are much, much more tractable than non-conformant/random behavior.

There is an analogous problem in distributed Information Retrieval (IR), e.g. 
text search.It is common for cooperating search servers to communicate Term 
Frequency (TF) and Document Frequency(DF) on each search server among cluster 
members so that distributed relevancy may be calculated (usually described as 
TF-IDF).With true Federated Search/Meta-search, this isn’t possible, and so 
search relevancy suffers.It has given federated search rather a black eye 
although there are many scholarly articles about it.

In the semantic web, the common solution is to offer both a download of TTL, 
RDF, or NTriples and have other SPARQL endpoints include your downloads, as 
Hugh Williams suggests.Although my semantic web site, 
http://id.nlm.nih.gov/mesh is a beta and likely to remain that way, we offer 
downloads of NTriples files at ftp://ftp.nlm.nih.gov/online/mesh.

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH



From: Hugh Williams [mailto:hwilli...@openlinksw.com]
Sent: Monday, November 16, 2015 8:59 AM
To: Eng.Bassam <bassa...@gmail.com<mailto:bassa...@gmail.com>>
Cc: 
virtuoso-users@lists.sourceforge.net<mailto:virtuoso-users@lists.sourceforge.net>
Subject: Re: [Virtuoso-users] federated SPARQL queries?

Hi Bassam,

I assume your ontology is already in RDF format, in which case see the 
following article on how to import/load and Ontology into Virtuoso:


http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtTipsAndTricksGuideImportOntology

or generally you can use any of the many methods of insert/load/import’ing RDF 
data into Virtuoso detailed at:


http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFInsert

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers

On 16 Nov 2015, at 07:48, Eng.Bassam 
<bassa...@gmail.com<mailto:bassa...@gmail.com>> wrote:

Hi
I'm Bassam a software engineer, I'm preparing master thesis in semantic web

I have my medical data in virtuoso and want to link my data to external 
ontology for diseases,
how can I do that :

must upload the ontology in virtuoso server, if yes how do that?

my objective is make federated SPARQL queries to traversal on my data also 
diseases ontology

thanks.

--
-

[Virtuoso-users] Parameterizing SPARQL in Virtuoso/PL

[Davis, Daniel (NIH/NLM) [C]]

I'm trying to implement a versioning policy our linked data, and when I see , 
located at http://id.nlm.nih.gov/mesh.   We currently load the triples each 
night, and we want to start being sensitive to which subjects no longer appear 
in the graph, updating the triples of subjects that exist in the new schema, 
and marking the subjects that do not exist in the old schema as "obsolete".

I've come up with a query that allows me to do some logic on specific subjects, 
but I need then to parameterize the copy of data.We'd prefer to do this in 
Virtuoso/PL, but I've not yet figured out how to do something like this:

CREATE PROCEDURE test_insert(in source_graph varchar, in target_graph varchar, 
in subject_uri varchar) {
  DECLARE target_iid iri_id;
  DECLARE subject_iid iri_id;

  target_iid := rdf_make_iid_of_qname(target_graph);
  subject_iid := rdf_make_iid_of_qname(subject_uri);

  DELETE FROM RDF_QUAD WHERE g = target_iid AND s = subject_iid;
  SPARQL INSERT { GRAPH target_graph { uri ?p ?o } } USING source_graph WHERE { 
uri ?p ?o };
}

Any help would be appreciated.

Thanks,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] Question on graph replication

[Davis, Daniel (NIH/NLM) [C]]

I want to confirm my assumption - is it true that the RDF Graph 
Replication 
feature is only available in virtuoso professional?
I was surprised to see that the stored procedures are there:

SQL> db.dba.rdf_repl_start();

Done. -- 8 msec.

Thanks

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Parameterizing SPARQL in Virtuoso/PL

[Davis, Daniel (NIH/NLM) [C]]

Hugh,

Thanks, using ?? is better binding that the solution I came up with so far, 
which was sprintf() and SPARQL_EVAL().   This will lead the graph 
<http://id.nlm.nih.gov/mesh> available on http://id.nlm.nih.gov/mesh/query (UI) 
and http://id.nlm.nih.gov/mesh/sparql (endpoint) to retain information about 
deleted supplementary concepts within the year and other deleted subjects 
year-over-year.

Thanks again,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

From: Hugh Williams [mailto:hwilli...@openlinksw.com]
Sent: Sunday, September 06, 2015 8:56 PM
To: Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov>
Cc: virtuoso-users@lists.sourceforge.net
Subject: Re: [Virtuoso-users] Parameterizing SPARQL in Virtuoso/PL

Hi Daniel,

Details on passing parameters in SPARQL queries can be found at:

  http://docs.openlinksw.com/virtuoso/rdfsparql.html#rdfsparqlinline

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers

On 4 Sep 2015, at 21:05, Davis, Daniel (NIH/NLM) [C] 
<daniel.da...@nih.gov<mailto:daniel.da...@nih.gov>> wrote:

I’m trying to implement a versioning policy our linked data, and when I see , 
located athttp://id.nlm.nih.gov/mesh.   We currently load the triples each 
night, and we want to start being sensitive to which subjects no longer appear 
in the graph, updating the triples of subjects that exist in the new schema, 
and marking the subjects that do not exist in the old schema as “obsolete”.

I’ve come up with a query that allows me to do some logic on specific subjects, 
but I need then to parameterize the copy of data.We’d prefer to do this in 
Virtuoso/PL, but I’ve not yet figured out how to do something like this:

CREATE PROCEDURE test_insert(in source_graph varchar, in target_graph varchar, 
in subject_uri varchar) {
  DECLARE target_iid iri_id;
  DECLARE subject_iid iri_id;

  target_iid := rdf_make_iid_of_qname(target_graph);
  subject_iid := rdf_make_iid_of_qname(subject_uri);

  DELETE FROM RDF_QUAD WHERE g = target_iid AND s = subject_iid;
  SPARQL INSERT { GRAPH target_graph { uri ?p ?o } } USING source_graph WHERE { 
uri ?p ?o };
}

Any help would be appreciated.

Thanks,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net<mailto:Virtuoso-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Vituoso API for generating triples from C++ program

[Davis, Daniel (NIH/NLM) [C]]

My experience is that you are better off loading the schema/vocabulary from 
Turtle (TTL) or RDF rather than OWL.   Aside from that, it does load the 
vocabulary/schema from RDF, and it does load the triples from RDF, and if you 
later add triples through a program, then you can export them all together as 
RDF.

For C++ I recommend ODBC.   I was able with little trouble to get perl 
DBD::ODBC to work with Virtuoso.   You can manipulate the RDF data using SQL 
statements or SPARQL statements over ODBC.As a query language, SPARQL is 
great, very expressive, and often more natural than SQL.   As a wire protocol, 
SPARQL is not there yet – ODBC/JDBC include not only a specification for 
cursors/paging but a rock-solid implementation.

Another thing you may want to consider is Virtuoso/PL – Virtuoso/PL can be 
developed much the way you work on a FORTH program – bottom-up coding one 
procedure at a time.   C++ can then call those procedures through ODBC.

Hoping this helps,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

From: Delaram Ra [mailto:delaram...@gmail.com]
Sent: Wednesday, September 16, 2015 3:40 PM
To: virtuoso-users@lists.sourceforge.net
Subject: [Virtuoso-users] Vituoso API for generating triples from C++ program

Hi,

I'm trying to write a C++ program to parse 100,000 files to parse them, extract 
some meta data and create an ontology from the information which will be kept 
in memory per file.
I've been refer to Virtuoso by my professor.
My question is that is there any Virtuoso API (or any other way) that I can 
generate my triples based on the RDF schema that I have so I don't have to 
write the syntax myself? (I'm looking for something like Jena framework for 
Java applications which allows you to load your rdf schema and add your data to 
it based on the schema and it will generate the whole rdf file for you with 
correct syntax)
It's a part of my thesis and there is a constraint from the people who will be 
using this library that keeps me from using any other programming environment 
other than C++.

Any help is much appreciated.

Regards,
Delaram
--
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] saving results to a file

[Davis, Daniel (NIH/NLM) [C]]

Thanks Hugh,

This is more or less what I'm now doing.Since we are not running the HTTP 
server in production, the latter option is not available to us.

-Original Message-
From: Hugh Williams [mailto:hwilli...@openlinksw.com] 
Sent: Friday, September 11, 2015 12:29 AM
To: Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov>
Cc: virtuoso-users@lists.sourceforge.net
Subject: Re: [Virtuoso-users] saving results to a file

Hi Daniel,

A query resultset from isql can be output to a file as detailed at:


http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtTipsAndTricksGuideISQLQueryResultOutput

Note you could also output results from a SPARQL query run against the /sparql 
endpoint to a file, using a curl command like the following I would expect:

curl --request POST 'http://127.0.0.1:8890/sparql/?' --header 
'Accept-Encoding: gzip' ' --data-urlencode 'query=define output:format "NT" 
CONSTRUCT { ?s ?p ?o } FROM <http://id.nlm.nih.gov/mesh/test> WHERE { ?s ?p ?o 
}' --output 'filename.gz'

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers

> On 9 Sep 2015, at 20:49, Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov> 
> wrote:
> 
> When I run the statement below in an ISQL command prompt, I get as a result a 
> BLOB of 225899 chars.
>  
> SPARQL define output:format "NT" CONSTRUCT { ?s ?p ?o } FROM 
> <http://id.nlm.nih.gov/mesh/test> WHERE { ?s ?p ?o };
>  
> I’m not quite sure how to use string_to_file() with a query like this…
>  
> Dan Davis, Systems/Applications Architect (Contractor), Office of 
> Computer and Communications Systems, National Library of Medicine, NIH
>  
> --
>  Monitor Your Dynamic Infrastructure at Any Scale With 
> Datadog!
> Get real-time metrics from all of your servers, apps and tools in one 
> place.
> SourceForge users - Click here to start your Free Trial of Datadog now!
> http://pubads.g.doubleclick.net/gampad/clk?id=241902991=/4140__
> _
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Automating RDF data imports in VIrtuoso

[Davis, Daniel (NIH/NLM) [C]]

> What is the best option for automating the import and update of RDFa/HTML 
> data on a regular basis into the virtuoso DB?

Jason, 

Are you running virtuoso on a Windows system or on a UNIX/Linux system?

In either case, I suggest you:
- Write a script that talks to the server on port  (e.g. the 
database port)
- Good choices are perl/python with ODBC or Java with JDBC are good 
options.
- Debug the script

If you are running this from UNIX/Linux, you can possibly edit a file to run 
regular jobs be entering:
crontab -e

If that doesn't work, you may have to ask your system administrators for help.
Entering the command 'man crontab' may help.



--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Automating RDF data imports in VIrtuoso

[Davis, Daniel (NIH/NLM) [C]]

Hi Paul,

We haven't had problem with data corruption, but the fix for the load memory 
leak is indeed a patch on top of 7.2.1.
If you download 7.2.1 and expand it, then you can apply the patch as follows:


- Take attached patch file and save to system

- Apply with Unix/Linux patch utility like this:

cd virtuoso-source-directory
patch -p1 < path-to-patch-file
./configure --with-readline -prefix path-to-install
make
make install

Our build is a little more complicated as we package it as an rpm.

As far as the memory leak, we are very certain it is gone - we have been 
monitoring the memory since and seen no leaks.

Thanks,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH


From: Hugh Williams [mailto:hwilli...@openlinksw.com]
Sent: Saturday, September 26, 2015 9:17 PM
To: Paul Houle 
Cc: virtuoso-users 
Subject: Re: [Virtuoso-users] Automating RDF data imports in VIrtuoso

Hi Paul,

If you a test case for recreating this data corruption issues I would suggest 
trying against the git develop/7 branch with all the latest fixes to see if it 
still persists ? Or if this step for recreation can be provide it we could test 
locally ?

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers

On 26 Sep 2015, at 23:39, Paul Houle 
> wrote:

I like the cloud solution of creating a new virtuoso system,  doing the load,  
having plenty of time to test it,  then replacing the production instance with 
the new instance and retiring the production instance.

The main advantage here is that there is no way a screw-up in the load 
procedure can trash the production system --  even if Virtuoso was entirely 
reliable,  as the data sources grow the rate of exceptional events (say you 
fill the disk) goes up.  The temporary server approach eliminates a lot of 
headaches and it is good cloud economics.  (if you run a server at AMZN for 1 
hour a day to update,  the cost of your system only goes up by %4).

I was having good luck with this approach until Virtuoso 7.2.0 came along and 
since then I've had problems similar in severity to what the N.I.H. was 
reporting,  it really looked like massive corruption of the data structures,  
7.2.1 did not help.

I don't know if these issues are fixed in the current TRUNK but if they are it 
would be nice to get an official release.

On Fri, Sep 25, 2015 at 1:31 PM, Haag, Jason 
> wrote:

Hi Users,

I'm trying to determine the best option for my situation for importing RDF data 
into Virtuoso. Here's my situation:

I currently have several RDF datasets available on my server. Each data set has 
an RDF dump available as RDF/XML, JSON-LD, and Turtle. These dumps are 
generated automatically without virtuoso from an HTML page marked up using RDFa.

What is the best option for automating the import of this data on a regular 
basis into the virtuoso DB? The datasets may grow so it should not just import 
the data once, but import on a regular basis, perhaps daily or weekly.

Based on what I've read in the documentation, this crawler option seems like 
the most appropriate option for my situation: 
http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtSetCrawlerJobsGuideDirectories

Can anyone verify if this would be the best approach? Does anyone know if the 
crawler supports RDFa/HTML or should it point to a specific directory with only 
the RDF dump files?

Thanks in advance!

J Haag

--

___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users



--
Paul Houle

Applying Schemas for Natural Language Processing, Distributed Systems, 
Classification and Text Mining and Data Lakes

(607) 539 6254paul.houle on Skype   
ontolo...@gmail.com

:BaseKB -- Query Freebase Data With SPARQL
http://basekb.com/gold/

Legal Entity Identifier Lookup
https://legalentityidentifier.info/lei/lookup/

Join our Data Lakes group on LinkedIn
https://www.linkedin.com/grp/home?gid=8267275

--
___
Virtuoso-users mailing list

[Virtuoso-users] virtuoso crash during nightly reload

[Davis, Daniel (NIH/NLM) [C]]

Virtuoso's load via TTLP of a N-triples file failed for me last night.   I am 
running 7.2.1 + the memory leak patch I posted about earlier.
The back-trace doesn't have that many symbols in it (optimized, stripped).

So, I'm skipping a lot:

...
11:20:25 /usr/nlm/virtuoso/bin/virtuoso-t(table_source_input+0x677) [0x61ae47]
...
11:20:25 /usr/nlm/virtuoso/bin/virtuoso-t(sf_sql_execute_w+0x8f) [0x621a2f]
...
11:20:25 /lib64/libc.so.6(clone+0x6d) [0x7f99eb2969ad]
11:20:25 GPF: cetop.c:1388 not supposed to hit lt rl ce if looking for end of 
range
...

If it crashes in clone(), does that look familiar to anyone?

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] saving results to a file

[Davis, Daniel (NIH/NLM) [C]]

When I run the statement below in an ISQL command prompt, I get as a result a 
BLOB of 225899 chars.

SPARQL define output:format "NT" CONSTRUCT { ?s ?p ?o } FROM 
 WHERE { ?s ?p ?o };

I'm not quite sure how to use string_to_file() with a query like this...

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
Monitor Your Dynamic Infrastructure at Any Scale With Datadog!
Get real-time metrics from all of your servers, apps and tools
in one place.
SourceForge users - Click here to start your Free Trial of Datadog now!
http://pubads.g.doubleclick.net/gampad/clk?id=241902991=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] deadlocks in bulk loading using rdf_loader_run function

[Davis, Daniel (NIH/NLM) [C]]

One way to achieve this is to use RDF Graph 
Groups
 and read-only users, e.g. users in the SPARQL_SELECT group.   The idea is to 
load data into a graph that your normal, read-only user will not see due to RDF 
Graph Security, and 
then move the triples into a graph that they can see.

Of course, the data is ultimately in RDF_QUAD.The advantage of this is that 
a more privileged user can access the data using SPARQL and SPARUL:

PREFIX …
INSERT { GRAPH  { ?s ?p ?o } }
WHERE { GRAPH  { ?s ?p ?o } }

This also allows for some transformation as data is copied.

Another advantage is that Virtuoso has assigned an IRI_ID to the triples, and 
so when changes are made, they can be as limited as possible (in SQL):

UPDATE RDF_QUAD
   SET g = rdf_make_iid_of_qname(newgraph_iri)
 WHERE g = rdf_make_iid_of_qname(oldgraph_iri);

Hope this helps,

Dan Davis

From: Gang Fu [mailto:gangfu1...@gmail.com]
Sent: Monday, December 21, 2015 9:28 AM
To: virtuoso-users 
Subject: Re: [Virtuoso-users] deadlocks in bulk loading using rdf_loader_run 
function

Is it possible to load files into staging tables first, then merge them into 
RDF_QUAD?

Best,
Gang

On Fri, Dec 18, 2015 at 2:34 PM, Gang Fu 
> wrote:
Hi,

I have deadlock problem with function 'rdf_loader_run' running in parallel, and 
I have tried many ways to avoid deadlocks, including

1)  Preprocess files using sort and uniq

2)  Loading without indexes

3)  Split big files into very small chunks

But deadlocks problem is still there. It seems the only way to avoid this is 
loading in single thread, which is very slow… any other solutions?

Best,
Gang


--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] output:format "NT" question

[Davis, Daniel (NIH/NLM) [C]]

Thanks Hugh,

Changing that just got me to a maximum size vector error.   Turns out there are 
many takes on saving this to files using http_nt_triple(), string_output(), 
string_to_file() through stored procedures, and that got me through.

The release of my URI preserving algorithm is Dec. 3rd, and after that I'll 
lurk until NLM makes a decision on whether to change this to production, and 
maybe add additional ontologies.

I wanted to take the opportunity to thank you for your patient support as I 
learned Virtuoso more deeply.   Its been a project; you've helped.

-Dan

From: Hugh Williams [hwilli...@openlinksw.com]
Sent: Saturday, November 21, 2015 8:37 PM
To: Davis, Daniel (NIH/NLM) [C]
Cc: virtuoso-users@lists.sourceforge.net
Subject: Re: [Virtuoso-users] output:format "NT" question

Hi Daniel,

A default virtuoso.ini file has a "ResultSetMaxRows = 1” in the "[SPARQL]" 
section  which I imagine  when using your initial script it was hitting this 
restriction on max size of a SPARQL result set ...

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers

> On 16 Nov 2015, at 19:11, Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov> 
> wrote:
>
> Guys,
>
> I still don’t know why it doesn’t work, but I’ve adapted 
> https://www.mail-archive.com/virtuoso-users@lists.sourceforge.net/msg03950.html
>  
> andhttp://joaorosilva.no-ip.org/wiki/doku.php/mainblog:dump_and_load_graphs_in_virtuoso
>  to my own purposes and they seem to work well.
>
> From: Davis, Daniel (NIH/NLM) [C]
> Sent: Monday, November 16, 2015 1:09 PM
> To: virtuoso-users@lists.sourceforge.net
> Subject: output:format "NT" question
>
> I have a simple script to export triples, but it seems not to get all 
> triples, but only about 10,000.I cannot guess what may be the problem.
> Any advice is appreciated.
>
> isql  dba “$PASSWORD” BANNER=OFF BLOBS=ON VERBOSE=OFF ECHO=OFF PROMPT=OFF 
> TIMEOUT=0 >fullmesh.nt < SPARQL
> define output:format "NT"
> PREFIX meshv: <http://id.nlm.nih.gov/mesh/vocab#>
> CONSTRUCT { ?s ?p ?o }
> WHERE {
>   GRAPH <http://id.nlm.nih.gov/mesh> {
> ?s ?p ?o
>   }
> };
> EOF
>
>
> Dan Davis, Systems/Applications Architect (Contractor),
> Office of Computer and Communications Systems,
> National Library of Medicine, NIH
>
> --
> Presto, an open source distributed SQL query engine for big data, initially
> developed by Facebook, enables you to easily query your data on Hadoop in a
> more interactive manner. Teradata is also now providing full enterprise
> support for Presto. Download a free open source copy now.
> http://pubads.g.doubleclick.net/gampad/clk?id=250295911=/4140___
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users


--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Replication on a schedule

[Davis, Daniel (NIH/NLM) [C]]

So, I'm not sure whether these are available in the Conductor interface, but 
the cluster control utility (clctl) seems to support what I want to do if I 
have scheduled jobs which turn on a cluster member for replication and then 
stop the replication at some point:

At 2 am, clctl rebuild 2
...
At 4 am, check that node 2 is up-to-date, and then disable it - 
clctl disable 2

This is probably an anti-pattern however.  Running with row-level autocommit 
has been good because I don't have clustering, and am manually replicating via 
scripts.  I should probably rewack my stored procedures and systems 
architecture so that my update jobs run efficiently with transaction isolation.

From: Davis, Daniel (NIH/NLM) [C]
Sent: Friday, June 10, 2016 10:55 AM
To: virtuoso-users <virtuoso-users@lists.sourceforge.net>
Subject: [Virtuoso-users] Replication on a schedule

Hi, does anyone out there with the Cluster edition of Virtuoso replicate on a 
schedule rather than as deltas are flushed?  Is this desirable?

The issue is that in a star topology, replication represents an increased risk 
of the replicant failing.  It may therefore be better to replicate only during 
certain time frames so that the cluster remains redundant.   I know I could do 
this scripts subscribing and subscribing to deltas, but it would be more 
hands-off if Virtuoso has a simple concept of when to apply deltas.

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] Replication on a schedule

[Davis, Daniel (NIH/NLM) [C]]

Hi, does anyone out there with the Cluster edition of Virtuoso replicate on a 
schedule rather than as deltas are flushed?  Is this desirable?

The issue is that in a star topology, replication represents an increased risk 
of the replicant failing.  It may therefore be better to replicate only during 
certain time frames so that the cluster remains redundant.   I know I could do 
this scripts subscribing and subscribing to deltas, but it would be more 
hands-off if Virtuoso has a simple concept of when to apply deltas.

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Replication on a schedule

[Davis, Daniel (NIH/NLM) [C]]

Thanks, Hugh.

I figured out how I could make it work, anyway.   Such "replication on a 
schedule" is not usually the way that database replication works, and I'm not 
sure I should try to make it work this way.   I may work with OpenLink to 
decide on that if we decide to purchase a license for Virtuoso Cluster - this 
email was a product of research towards that decision.

-Original Message-
From: Hugh Williams [mailto:hwilli...@openlinksw.com] 
Sent: Friday, June 10, 2016 5:36 PM
To: Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov>
Cc: virtuoso-users <virtuoso-users@lists.sourceforge.net>
Subject: Re: [Virtuoso-users] Replication on a schedule

Hi Daniel,

This is an open source mailing list for which cluster is not available as it is 
a commercial only feature, thus not likely to get much response on it …

Note sure what you mean by "replicate on a schedule rather than as deltas are 
flushed” ?

You seem to be seeking to use Virtuoso scale out Elastic Cluster [1][2] which 
is for horizontal scaling across multiple machine to pool resources (storage, 
memory, cores) for hosting large datasets that could not otherwise be hosting 
on a single machine. Virtuoso also has normal Database replication although 
this is a commercial feature only also and is also used for providing Virtuoso 
RDF Graph Replication clustering[3] of of single or scale out cluster farms. 
Based on the email you sent to support is would appear the Virtuoso RDF Graph 
Replication Clustering is most suitable for you use case.

See also the different cluster configuration architectures at [4] …

[1] http://docs.openlinksw.com/virtuoso/clusteroperation.html
[2] http://docs.openlinksw.com/virtuoso/clusterstcnf.html
[3] http://docs.openlinksw.com/virtuoso/rdfgraphreplication.html
[4] 
http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtClusteringDiagrams

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers

> On 10 Jun 2016, at 17:17, Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov> 
> wrote:
> 
> So, I’m not sure whether these are available in the Conductor interface, but 
> the cluster control utility (clctl) seems to support what I want to do if I 
> have scheduled jobs which turn on a cluster member for replication and then 
> stop the replication at some point:
>  
> At 2 am, clctl rebuild 2
> …
> At 4 am, check that node 2 is up-to-date, and then disable it - 
> clctl disable 2
>  
> This is probably an anti-pattern however.  Running with row-level autocommit 
> has been good because I don’t have clustering, and am manually replicating 
> via scripts.  I should probably rewack my stored procedures and systems 
> architecture so that my update jobs run efficiently with transaction 
> isolation.
>  
> From: Davis, Daniel (NIH/NLM) [C] 
> Sent: Friday, June 10, 2016 10:55 AM
> To: virtuoso-users <virtuoso-users@lists.sourceforge.net>
> Subject: [Virtuoso-users] Replication on a schedule
>  
> Hi, does anyone out there with the Cluster edition of Virtuoso replicate on a 
> schedule rather than as deltas are flushed?  Is this desirable?
>  
> The issue is that in a star topology, replication represents an increased 
> risk of the replicant failing.  It may therefore be better to replicate only 
> during certain time frames so that the cluster remains redundant.   I know I 
> could do this scripts subscribing and subscribing to deltas, but it would be 
> more hands-off if Virtuoso has a simple concept of when to apply deltas.
>  
> Dan Davis, Systems/Applications Architect (Contractor),
> Office of Computer and Communications Systems,
> National Library of Medicine, NIH
>  
> --
> What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
> patterns at an interface-level. Reveals which users, apps, and protocols are 
> consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
> J-Flow, sFlow and other flows. Make informed decisions using capacity 
> planning reports. 
> https://ad.doubleclick.net/ddm/clk/305295220;132659582;e___
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users



smime.p7s
Description: S/MIME cryptographic signature

[Virtuoso-users] Restrict dba login to localhost

[Davis, Daniel (NIH/NLM) [C]]

Is there any way to restrict DBA login to requests originating from localhost?  
  We have an application that would like to use the JDBC interface, and we've 
exposed port  on the network, but we are a little paranoid about security 
and want to restrict DBA login to the localhost.

In our environment, we haven't turned off conductor and the web server.

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] Making REGEX filters fast

[Davis, Daniel (NIH/NLM) [C]]

Is there a way to make FILTER(REGEX(?label, 'Insurance, Health/utilization', 
'I')) queries fast?

I can see that REGEX is going to require a linear scan of the data, and that 
there are stored functions that do the matching.

Will STRSTARTS be able to work quickly?   Does the database collation allow 
case insensitive matching with STRSTARTS?

Does one require a license for Virtuoso to use Full-text bif:contains (which is 
not quite the same as REGEX, of course)?

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] uploade TTl big file?

[Davis, Daniel (NIH/NLM) [C]]

My experience was that I needed explicit SQL:

   DELETE FROM load_list;

I didn't see anything else.   I also didn't experience any acceleration from 
TTLP_MT vs TTLP even though I had multiple cores.   I only had 2 cores, and so 
I don't know whether it reflects on the product.

From: Eng.Bassam [bassa...@gmail.com]
Sent: Monday, January 25, 2016 12:21 PM
To: Hugh Williams; virtuoso-users@lists.sourceforge.net
Subject: Re: [Virtuoso-users] uploade TTl big file?

I have split my file to 4 files, and Now I'm trying bulk loader strategy,
I see after create the loader, I can see the files handled
select * from DB.DBA.load_list;

the first time was an error in my files, how can I try again run loader  and 
check status in DB.DBA.load_list (delete last result)??

thanks

On Mon, Jan 25, 2016 at 2:20 PM, Hugh Williams 
> wrote:
Hi Bassam,

You can call the TTLP_MT() function as you are doing for a single file, but 
generally we would recommend the use of the Virtuoso RDF Bulk loader as 
detailed at:

http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtBulkRDFLoader

Which enable you to better manage the bulk loading of RDF datasets.

The Virtuoso LDMeter functions can also be used to monitor the progress of data 
loads ie load rates etc as detailed at:

http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtTipsAndTricksGuideLDMeterUtility

Generally you need to ensure you have tuned the Virtuoso server for RDF 
performance, as if you run out of Buffers (memory) and have to swapping to and 
from disk, this will kill load rates and performance generally as detailed at:

http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFPerformanceTuning

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers



On 25 Jan 2016, at 10:45, Eng.Bassam 
> wrote:

Hello

I have a big TTL file about (2GB), I try upload it in virtuoso via ISQL:

DB.DBA.TTLP_MT (file_to_string_output ('C:/Program Files/OpenLink 
Software/Virtuoso 7.2/vad/geneDiseaseAssociation.ttl'), '', 
'http://localhost:8890/ontologies/DisGen');

is this right, or wrong?
I need fastest way, and wish to see a progress bar to know how remaining time

note:
while it executing, I made COUNT(*) via SPARQL to see if count of added triples 
to graph increase or not, I think after 1 hour not add any triple on graph and 
ISQL still busy :(

--
---
> Eng.Bassam Najeeb.
>Software Engineer.

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users




--
---
> Eng.Bassam Najeeb.
>Software Engineer.


--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311=/4140
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Temporary DB size and server restart

[Davis, Daniel (NIH/NLM) [C]]

It looks like this is merely a cache, and it's OK for it to be bigger.It 
doesn't look like it has anything to do with my Virtuoso shutdown issues.

From: Davis, Daniel (NIH/NLM) [C]
Sent: Wednesday, March 30, 2016 12:09 PM
To: virtuoso-users <virtuoso-users@lists.sourceforge.net>
Subject: Temporary DB size and server restart

I have a strange failure condition somewhere in my deployment.Virtuoso goes 
down, and whenever I restart it, I see something like this in the log:

Unlinked the temp db file /usr/nlm/virtuoso/db/virtuoso-temp.db as its size 
(8180MB) was greater than TempDBSize INI (10MB)

Now, that is a big difference.Why would the virtuoso-temp DB size get to be 
so much bigger than the limit?What removes it?

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785471=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] System setup guidelines

[Davis, Daniel (NIH/NLM) [C]]

Maybe I'm missing something, but is there any place in the documentation I 
simple list of the things you should do to setup Virtuoso:

*Don't use NFS

*If you do use NFS, don't use soft mounts

*The change to add "vm.swappiness = 10" to /etc/sysctl.conf that is 
mentioned in the RDF performance tuning chapter.

*Any changes to limits, e.g. setrlimit/ulimit for the virtuoso user?

Thanks,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] System setup guidelines

[Davis, Daniel (NIH/NLM) [C]]

Thanks, Hugh.


From: Hugh Williams [hwilli...@openlinksw.com]
Sent: Wednesday, April 06, 2016 7:26 PM
To: Davis, Daniel (NIH/NLM) [C]
Cc: virtuoso-users
Subject: Re: [Virtuoso-users] System setup guidelines

Hi Dan,

The main places Virtuoso setup and tuning information can be found are:

http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ptune
http://docs.openlinksw.com/virtuoso/rdfperformancetuning.html

I saw the git issue and commit you made regarding the NFS issue you encountered 
which we shall be reviewing.

The only issue I have come across with NFS was with regards to file locking 
required by the virtuoso.lck, thus an INI file param is provided to specify an 
alternative location for it as detailed at:

http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ini_Database

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers



> On 6 Apr 2016, at 19:57, Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov> 
> wrote:
>
> Maybe I’m missing something, but is there any place in the documentation I 
> simple list of the things you should do to setup Virtuoso:
> ·Don’t use NFS
> ·If you do use NFS, don’t use soft mounts
> ·The change to add “vm.swappiness = 10” to /etc/sysctl.conf that is 
> mentioned in the RDF performance tuning chapter.
> ·Any changes to limits, e.g. setrlimit/ulimit for the virtuoso user?
>
> Thanks,
>
> Dan Davis, Systems/Applications Architect (Contractor),
> Office of Computer and Communications Systems,
> National Library of Medicine, NIH
>
> --
> ___
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users


--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] Temporary DB size and server restart

[Davis, Daniel (NIH/NLM) [C]]

I have a strange failure condition somewhere in my deployment.Virtuoso goes 
down, and whenever I restart it, I see something like this in the log:

Unlinked the temp db file /usr/nlm/virtuoso/db/virtuoso-temp.db as its size 
(8180MB) was greater than TempDBSize INI (10MB)

Now, that is a big difference.Why would the virtuoso-temp DB size get to be 
so much bigger than the limit?What removes it?

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785471=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Temporary DB size and server restart

[Davis, Daniel (NIH/NLM) [C]]

Virtuoso was terminated by the out-of-memory killer.   Most of my Virtuoso 
nodes had thread cleanup and resource cleanup set to 1, but not the one that 
failed.   So, this was a case of me not changing ThreadCleanupInterval and 
ResourceCleanupInterval when I installed this node.   I don’t trust my install 
script enough, and most people use puppet/chef or other tools.



From: Hugh Williams [mailto:hwilli...@openlinksw.com]
Sent: Wednesday, March 30, 2016 4:53 PM
To: Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov>
Cc: virtuoso-users <virtuoso-users@lists.sourceforge.net>
Subject: Re: [Virtuoso-users] Temporary DB size and server restart



Hi Daniel,



As indicated in the docs at 
http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ex_threadcleanupinterval
 :



*   TempDBSize

Controls the acceptable size of the temp database file. If on startup it's size 
(in MB) is greater than TempDBSize the file gets deleted and reset. This 
feature can be turned off by setting TempDBSize to 0. Note that the temp db 
file serves as an optimization storage only and doesn't have any client data 
that are not in either the main database files or the corresponding transaction 
log files.

If you Virtuoso database shutdown unexpectedly then the TempDB in use would not 
have been clean up which is what the message is indicated you should the 
consider manually removing as it contains no required data.



Thus you should focus on determining why the Virtuoso server shutdown 
unexpectedly, thus is anything of interest in the log prior to it shutting down 
or in the system log files that might indicate the cause of it shutdown ?



Best Regards

Hugh Williams

Professional Services

OpenLink Software, Inc.  //  http://www.openlinksw.com/

Weblog   -- http://www.openlinksw.com/blogs/

LinkedIn -- http://www.linkedin.com/company/openlink-software/

Twitter  -- http://twitter.com/OpenLink

Google+  -- http://plus.google.com/100570109519069333827/

Facebook -- http://www.facebook.com/OpenLinkSoftware

Universal Data Access, Integration, and Management Technology Providers







   On 30 Mar 2016, at 17:18, Davis, Daniel (NIH/NLM) [C] 
<daniel.da...@nih.gov<mailto:daniel.da...@nih.gov>> wrote:



   It looks like this is merely a cache, and it’s OK for it to be bigger.It 
doesn’t look like it has anything to do with my Virtuoso shutdown issues.



   From: Davis, Daniel (NIH/NLM) [C]
   Sent: Wednesday, March 30, 2016 12:09 PM
   To: virtuoso-users 
<virtuoso-users@lists.sourceforge.net<mailto:virtuoso-users@lists.sourceforge.net>>
   Subject: Temporary DB size and server restart



   I have a strange failure condition somewhere in my deployment.Virtuoso 
goes down, and whenever I restart it, I see something like this in the log:



   Unlinked the temp db file /usr/nlm/virtuoso/db/virtuoso-temp.db as its size 
(8180MB) was greater than TempDBSize INI (10MB)



   Now, that is a big difference.Why would the virtuoso-temp DB size get to 
be so much bigger than the limit?What removes it?



   Dan Davis, Systems/Applications Architect (Contractor),

   Office of Computer and Communications Systems,

   National Library of Medicine, NIH



   
--
   Transform Data into Opportunity.
   Accelerate data analysis in your applications with
   Intel Data Analytics Acceleration Library.
   Click to learn more.
   
http://pubads.g.doubleclick.net/gampad/clk?id=278785471=/4140___
   Virtuoso-users mailing list
   
Virtuoso-users@lists.sourceforge.net<mailto:Virtuoso-users@lists.sourceforge.net>
   https://lists.sourceforge.net/lists/listinfo/virtuoso-users



--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785471=/4140___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] CREATE TABLE IF NOT EXISTS

[Davis, Daniel (NIH/NLM) [C]]

What is the best way to run some SQL statements to create tables if they do not 
exists?
I'm trying to achieve (at last), true automated deployment.

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] Debugging JDBC Virtuoso connection

[Davis, Daniel (NIH/NLM) [C]]

I've done my UI using Java Spring, but so far, I've been doing it as an old 
dinosaur, running my build and my changes on the Linux host where Virtuoso is 
running.
Now, I'm trying to use Virtuoso remotely from another host.

>From Python, I can connect to port , but when I try to connect to Virtuoso 
>from Java, it does not seem to be working.

I would prefer to enable some logging from Virtuoso JDBC to see what the 
problem is - it is throwing connection refused, but the connection should be 
open.
What is it actually trying to connect with?

Thanks,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. http://sdm.link/zohodev2dev___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Debugging JDBC Virtuoso connection

[Davis, Daniel (NIH/NLM) [C]]

Hugh, 

I figured it out.   I used interactive Python to prove the port was open, and 
then discovered my own web app was overriding the serverName I set in the 
Tomcat resource.


From: Hugh Williams [hwilli...@openlinksw.com]
Sent: Thursday, August 11, 2016 9:47 AM
To: Davis, Daniel (NIH/NLM) [C]
Cc: virtuoso-users
Subject: Re: [Virtuoso-users] Debugging JDBC Virtuoso connection

Hi Daniel,

So the same application using the same JDBC connect string work on localhost 
where Virtuoso runs, but not from remote client ?

I once had same problem when java runtime was trying to make ipv6 connection 
from remote host and was getting “network is unreachable” type error and had to 
disable ivp6 on the  localhost machine Virtuoso was running on. So wondering if 
you might be having similar problem ? I trace the cause with strace using 
command like:

  strace -f -v -o  -s 1500 java -cp "virtjdbc4.jar:.” 
sample.java

which showed something like:

10738 connect(8, {sa_family=AF_INET6, sin6_port=htons(1168), inet_pton(AF_INET6,
 ":::127.0.0.1", _addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EN
ETUNREACH (Network is unreachable)

BTW, I assume you JDBC connect string is using the actual machine hostname or 
IP address and not “localhost”  from when is was just being run locally ?

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers

On 10 Aug 2016, at 16:48, Davis, Daniel (NIH/NLM) [C] 
<daniel.da...@nih.gov<mailto:daniel.da...@nih.gov>> wrote:

I’ve done my UI using Java Spring, but so far, I’ve been doing it as an old 
dinosaur, running my build and my changes on the Linux host where Virtuoso is 
running.
Now, I’m trying to use Virtuoso remotely from another host.

>From Python, I can connect to port , but when I try to connect to Virtuoso 
>from Java, it does not seem to be working.

I would prefer to enable some logging from Virtuoso JDBC to see what the 
problem is – it is throwing connection refused, but the connection should be 
open.
What is it actually trying to connect with?

Thanks,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning reports. 
http://sdm.link/zohodev2dev___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net<mailto:Virtuoso-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/virtuoso-users


--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. http://sdm.link/zohodev2dev
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] Release memory without restart

[Davis, Daniel (NIH/NLM) [C]]

Virtuoso-opensource in my environment is using 79% - 83% of memory after 
several weeks of nightly TTLP and daily queries.
When I restart it, it is using 26% of memory (on a 16GB machine).

I cannot restart it in my nightly data processing jobs, because that requires 
root, and my jobs do not run as root.

Is there a way I can synchronously cause it to release unused pages on demand, 
maybe after connecting via isql as admin?

Thanks,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports.http://sdm.link/zohodev2dev___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Deleting large number of triples

[Davis, Daniel (NIH/NLM) [C]]

So, this has nothing to do with the large vector size, but just to be sure the 
SPARQL is correct - do you wish to delete the subjects (and all their triples) 
where the subject has the predicate, or just the predicate itself?

As far as avoiding the maximum vector size, I think your best approach is to 
limit the number of matches and repeat the query until there are no results, 
maybe with a count query in-between.   I have had to do similar sorts of 
work-arounds to avoid the maximum # of results and maximum size of string 
issues.   For instance, my first attempts to export large NTriples files after 
processing failed due to these issues.   You may be able to adapt the code 
below, but I think that a repeated deleted query limited to a # of triples will 
be best in your case.

Anyway, the code:

CREATE PROCEDURE meshrdf_export(in graph_uri varchar, in file_name varchar) {
DECLARE banner any;
DECLARE env, ses any;
DECLARE ses_len, max_ses_len any;

SET isolation = 'uncommitted';

max_ses_len := 1000;

--
-- Truncate file and write a comment line indicating the graph and datetime 
of export.
--
--no_c_escapes-
banner := sprintf('# <%s> exported at %s\n', graph_uri, datestring(now()));
string_to_file (file_name, banner, -2);

env := vector (0, 0, 0);
ses := string_output ();

FOR (SELECT * FROM (SPARQL
define input:storage ""
SELECT ?s ?p ?o WHERE {
  GRAPH `iri(?:graph_uri)` {
?s ?p ?o
  }
} ORDER BY ?s ?p ?o) AS sub OPTION (loop)) DO {
http_nt_triple (env, "s", "p", "o", ses);
ses_len := length (ses);

IF (ses_len > max_ses_len) {
string_to_file (file_name, ses, -1);
ses := string_output ();
}
}
IF (length (ses)) {
string_to_file (file_name, ses, -1);
}
}

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH


From: Pantelis Natsiavas [mailto:natsia...@gmail.com]
Sent: Wednesday, August 17, 2016 4:36 AM
To: virtuoso-users 
Subject: [Virtuoso-users] Deleting large number of triples

Hi everybody.

I am trying to delete a large number of triples of a very big graph. The graph 
contains 217.609.545 triples and I want to delete all the triples having a 
specific predicate (64.884.016 triples).

I am trying to do it through the isql-v command line interface, using the 
command:

SPARQL DEFINE sql:log-enable 3
WITH 
DELETE { ?s  ?o }
WHERE{ ?s  ?o }

After some time (I don't know exactly how much) I got the error

*** Error 42000: [Virtuoso Driver][Virtuoso Server]FRVEC: array in for vectored 
over max vector length 200 > 100
at line 1 of Top-Level:

 I checked the virtuoso.log and I see nothing related to the specific error.

I changed the parameters in virtuoso.ini:
MaxQueryMem  = 8G  ; from 2G
VectorSize = 1000   ; not changed
MaxVectorSize = 200  ; from 100
AdjustVectorSize = 1   ; from 0

I am not very confident about these changes in virtuoso settings, but checking 
the http://docs.openlinksw.com/virtuoso/dbadm.html these changes seemed the 
right thing to do.

I restarted the VM and retried the whole process. After one hour, the memory 
consumed by Virtuoso got around 100% and got an error:
*** Error 08S01: [Virtuoso Driver]CL065: Lost connection to server

Please note that from previous similar errors, I already have the following 
virtuoso.ini settings:
NumberOfBuffers = 136
MaxDirtyBuffers = 100
ThreadCleanupInterval= 1
ResourcesCleanupInterval = 1

My questions:
1. Is there any way to improve my query in order to facilitate its processing? 
It is the first time I am doing a DELETE query and I am not comfortable with it.
2. Is there any way to "split" the query so that it doesn't need to handle all 
these triples at once?
3. Alternatively, is there any configuration change that might improve memory 
handling in order to handle such big queries?

Kind regards,
Pantelis Natsiavas


--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] Understanding long start time

[Davis, Daniel (NIH/NLM) [C]]

So, I have virtuoso, and I spend some time loading lots of ntriples files into 
it with TTLP, and then I wait for 2 minutes for checkpoint to complete, with 
CheckpointInterval = 60.   Then, I restart it with +wait.It still has lots 
of I/O to do, doing a roll forward.

The Database Administration section, 
http://docs.openlinksw.com/virtuoso/dbadm/, does not define the units for 
checkpoint interval - can I assume that I would have needed to wait 60 minutes 
to assure that a checkpoint had been taken and no roll-forward would be 
necessary?

If not, can someone please explain what I can do to assure that I/O needed is 
done in a timely fashion.   I used to have the following in my load code, but I 
took it out to avoid a crash that may or may not have been related:

Checkpoint_interval(-1);
CHECKPOINT;
Checkpoint_interval(60);

Thanks,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] Scope of log_enable and effect on deletes

[Davis, Daniel (NIH/NLM) [C]]

I am not sure I understand the scope of the log_enable() procedure.If I 
call log_enable(2,1); in a stored procedure, I am sure that it will affect the 
entire connection, but, will it affect other connections as well.   Would it, 
for instance, persist until reboot and affect other connections?   How can I 
verify this - is there any command that will output the current transaction 
isolation level?

Also, I have an inexplicable situation where my stored procedure loads a 
dataset in NTriples format into a graph, and then does some processing on that 
dataset.   It is important for the processing that the graph be empty at the 
beginning, e.g. before the load.   Try as I might, I seem to end-up in 
situations where the graph is not-empty, and I believe that this may be some 
interaction between log_enable() and the delete statement.   I would think 
naively that log_enable(2,1); would have no effect on the serialization of 
row-level operations *in the same connection*, but perhaps I misunderstand.

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Scope of log_enable and effect on deletes

[Davis, Daniel (NIH/NLM) [C]]

Thanks, Quentin.

Then it is not the log setting.  When I try to trouble-shoot in a new 
connection (isql rather than perl DBI over ODBC), I observe very odd results 
indeed:

SQL> sparql select count(*) from <http://id.nlm.nih.gov/mesh/updates> where { 
?s a ?o };
…
4393
SQL> sparql clear graph <http://id.nlm.nih.gov/mesh/updates>;
…
SQL> sparql select count(*) from <http://id.nlm.nih.gov/mesh/updates> where { 
?s a ?o };
…
4

If no delete has ever been done, the number of subjects remaining should almost 
2 million – 1994719.


From: Quentin [mailto:quentin@guidinghand.solutions]
Sent: Tuesday, February 21, 2017 1:41 PM
To: virtuoso-users@lists.sourceforge.net
Subject: Re: [Virtuoso-users] Scope of log_enable and effect on deletes


From the manual:

>While log on and off setting alone is reset at the end of the transaction so 
>that one does not end up with logging disabled by accident, the row-by-row 
>autocommit mode causes the setting to be permanent inside the calling 
>connection or web request. That is, for a SQL client the setting stays in 
>effect until changed or disconnected and for a web request it stays in effect 
>until the request is completed.

http://docs.openlinksw.com/virtuoso/fn_log_enable/



If you want to determine whether the log is still disabled at a point in time 
then this is one round-about way of doing it:

>Attempt to disable transaction log that is disabled already result in an 
>error. If the second parameter is passed and it is not equal to zero then the 
>error is suppressed (and it's still possible to turn on or off autocommit).

Set an exception handler to catch that error and set a flag to indicate that 
the log was previously disabled.  If there's no error/flag set then proceed to 
re-enable the log.



There might be better ways of doing it but that should work.

--
Regards,

Quentin Serrurier.

GuidingHand.Solutions

On 2017-02-22 02:01, Davis, Daniel (NIH/NLM) [C] wrote:
I am not sure I understand the scope of the log_enable() procedure.If I 
call log_enable(2,1); in a stored procedure, I am sure that it will affect the 
entire connection, but, will it affect other connections as well.   Would it, 
for instance, persist until reboot and affect other connections?   How can I 
verify this – is there any command that will output the current transaction 
isolation level?

Also, I have an inexplicable situation where my stored procedure loads a 
dataset in NTriples format into a graph, and then does some processing on that 
dataset.   It is important for the processing that the graph be empty at the 
beginning, e.g. before the load.   Try as I might, I seem to end-up in 
situations where the graph is not-empty, and I believe that this may be some 
interaction between log_enable() and the delete statement.   I would think 
naively that log_enable(2,1); would have no effect on the serialization of 
row-level operations *in the same connection*, but perhaps I misunderstand.

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH



--

Check out the vibrant tech community on one of the world's most

engaging tech sites, SlashDot.org! http://sdm.link/slashdot


___

Virtuoso-users mailing list

Virtuoso-users@lists.sourceforge.net<mailto:Virtuoso-users@lists.sourceforge.net>

https://lists.sourceforge.net/lists/listinfo/virtuoso-users
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] Java Melody with Virtuoso

[Davis, Daniel (NIH/NLM) [C]]

Has anyone out there done a plugin so that Java Melody can monitor Virtuoso 
JDBC connections?
I have a semantic web application using Virtuoso behind Jena behind Spring, and 
I have Java Melody monitoring it.

It would be great to have Melody collect information on queries, but I'm not 
likely to implement that myself.
Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Deleting large number of triples

[Davis, Daniel (NIH/NLM) [C]]

The query seems fine.   The reason it doesn’t work is that Virtuoso first loads 
the triples that match into a vector (which has a limited size), and then 
deletes those that match the delete clause.   In this case, because your 
problem seems simple, you can just drop to SQL to avoid that:

LOG_ENABLE(3,0);
DELETE FROM RDF_QUAD
   WHERE g = rdf_make_iid_of_qname(graph_uri)
AND p = rdf_make_iid_of_qname(predicate_uri);

If you had wanted to delete all the triples for subjects that match, then 
SPARQL delete offers some powerful abstractions, but Virtuoso’s vector size 
would still get in your way.   It would be nice if the buffering were 
automatic, but I’m not sure how that would work given the delete’s capability 
of being an entirely different set of triples related to the others – there are 
even BIND and other SPARQL capabilities that OpenLink’s engineers would have to 
deal with.

From: Pantelis Natsiavas [mailto:natsia...@gmail.com]
Sent: Wednesday, August 17, 2016 10:55 AM
To: Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov>
Cc: virtuoso-users <virtuoso-users@lists.sourceforge.net>
Subject: Re: [Virtuoso-users] Deleting large number of triples

Thank you for your advice Daniel.

Actually I want to delete only the statements containing the specific 
predicate. I don't want to delete all the triples containing the subject of the 
predicate. As I have already said, I don't feel comfortable with the DELETE 
queries.
Is my query wrong? Could you suggest the correct query?

Kind regards,
Pantelis Natsiavas

2016-08-17 17:19 GMT+03:00 Davis, Daniel (NIH/NLM) [C] 
<daniel.da...@nih.gov<mailto:daniel.da...@nih.gov>>:
So, this has nothing to do with the large vector size, but just to be sure the 
SPARQL is correct - do you wish to delete the subjects (and all their triples) 
where the subject has the predicate, or just the predicate itself?

As far as avoiding the maximum vector size, I think your best approach is to 
limit the number of matches and repeat the query until there are no results, 
maybe with a count query in-between.   I have had to do similar sorts of 
work-arounds to avoid the maximum # of results and maximum size of string 
issues.   For instance, my first attempts to export large NTriples files after 
processing failed due to these issues.   You may be able to adapt the code 
below, but I think that a repeated deleted query limited to a # of triples will 
be best in your case.

Anyway, the code:

CREATE PROCEDURE meshrdf_export(in graph_uri varchar, in file_name varchar) {
DECLARE banner any;
DECLARE env, ses any;
DECLARE ses_len, max_ses_len any;

SET isolation = 'uncommitted';

max_ses_len := 1000;

--
-- Truncate file and write a comment line indicating the graph and datetime 
of export.
--
--no_c_escapes-
banner := sprintf('# <%s> exported at %s\n', graph_uri, datestring(now()));
string_to_file (file_name, banner, -2);

env := vector (0, 0, 0);
ses := string_output ();

FOR (SELECT * FROM (SPARQL
define input:storage ""
SELECT ?s ?p ?o WHERE {
  GRAPH `iri(?:graph_uri)` {
?s ?p ?o
  }
} ORDER BY ?s ?p ?o) AS sub OPTION (loop)) DO {
http_nt_triple (env, "s", "p", "o", ses);
ses_len := length (ses);

IF (ses_len > max_ses_len) {
string_to_file (file_name, ses, -1);
ses := string_output ();
}
}
IF (length (ses)) {
string_to_file (file_name, ses, -1);
}
}

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH


From: Pantelis Natsiavas 
[mailto:natsia...@gmail.com<mailto:natsia...@gmail.com>]
Sent: Wednesday, August 17, 2016 4:36 AM
To: virtuoso-users 
<virtuoso-users@lists.sourceforge.net<mailto:virtuoso-users@lists.sourceforge.net>>
Subject: [Virtuoso-users] Deleting large number of triples

Hi everybody.

I am trying to delete a large number of triples of a very big graph. The graph 
contains 217.609.545 triples and I want to delete all the triples having a 
specific predicate (64.884.016 triples).

I am trying to do it through the isql-v command line interface, using the 
command:

SPARQL DEFINE sql:log-enable 3
WITH 
DELETE { ?s  ?o }
WHERE{ ?s  ?o }

After some time (I don't know exactly how much) I got the error

*** Error 42000: [Virtuoso Driver][Virtuoso Server]FRVEC: array in for vectored 
over max vector length 200 > 100
at line 1 of Top-Level:

 I checked the virtuoso.log and I see nothing related to the specific error.

I changed the parameters in virtuoso.ini:
MaxQueryMem  = 8G  ; from 2G
VectorSize = 1000   ; not changed
MaxVectorSize = 200  ; from 100
AdjustVecto

Re: [Virtuoso-users] Out of curiosity, some questions about OpenSource Virtuoso settings

[Davis, Daniel (NIH/NLM) [C]]

Tangentially on JDBC/ODBC vs HTTP.HTTP is slow because of the transport, 
but JDBC/ODBC has more mature models of paging over results, pre-fetching, etc. 
  I feel that RDF is mature as a Data format, but as a protocol it is not that 
mature.

-Original Message-
From: Hugh Williams [mailto:hwilli...@openlinksw.com] 
Sent: Monday, September 12, 2016 5:30 AM
To: Lorenz Buehmann 
Cc: virtuoso-users@lists.sourceforge.net
Subject: Re: [Virtuoso-users] Out of curiosity, some questions about OpenSource 
Virtuoso settings

Hi Lorenz,

In Virtuoso 7+  Vectored query execution [1] enables single, typically complex 
analytical types queries, to be broken down and executed on multiple threads. 
The INI file param that controls this is "ThreadsPerQuery" [2] , which controls 
the maximum number of threads that can be claimed from the thread pool by a 
single query, and there are other associated params details at [2] .

[1] http://docs.openlinksw.com/virtuoso/vexqrparl.html
[2] http://docs.openlinksw.com/virtuoso/vexqrparlconfp/

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers

> On 12 Sep 2016, at 07:13, Lorenz Buehmann 
>  wrote:
> 
> Hi,
> 
> just as a follow-up question:
> 
> I know it supports inter-query parallelization, but does it also support 
> intra-query parallelization, i.e. using multiple threads to compute the 
> result of a single query? If yes, which parameter is used to configure this?
> 
> 
> Cheers,
> 
> Lorenz
> 
> On 12.09.2016 05:57, Kingsley Idehen wrote:
>> On 9/10/16 10:41 PM, giacom...@libero.it
>>  wrote:
>> 
>>> So, I have a bunch of questions about Virtuoso.
>>> 
>>> * Are SPARQL queries performed concurrently (using the standard 
>>> virtuoso.ini configuration)?
>>> 
>> Yes, Virtuoso is highly multi-threaded. In addition, it has 
>> vectorized query execution i.e., many query batches per thread, handled 
>> concurrently .
>> 
>>> * Are Virtuoso transactions autocommittable, or there aren't any 
>>> kind of transactions when connecting to the ODBC driver?
>>> 
>> Depends, by default you have Read Committed Isolation level.
>> 
>>> * Does Virtuoso automatically performs triple indexing before query data? 
>>> 
>> No, it has indexes in place.
>> 
>> 
>>> In
>>> some systems I have to configure it manually.
>>> * Does the usage of ODBC degradates the benchmarking of Virtuoso's 
>>> SPARQL query or the time required to store the triples within a given named 
>>> graph?
>>> 
>> No, if anything that's faster than HTTP.
>> 
>> Links:
>> 
>> [1]
>> http://docs.openlinksw.com/virtuoso/isolation/
>> 
>> [2]
>> 
>> http://wikis.openlinksw.com/VirtuosoWikiWeb/ChangeVirtuosoSDefaultTra
>> nsactionIsolationLevel
>> 
>> [3]
>> http://docs.openlinksw.com/virtuoso/fn_log_enable.html
>> 
>> [4]
>> 
>> http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFPer
>> formanceTuning
>> 
>> 
>> [5]
>> 
>> http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtBulkRD
>> FLoader#Running
>> 
>> multiple Loaders
>> 
>> 
>> Kingsley
>> 
>> 
>>> Thanks in advance for any support,
>>> 
 JackB
 
>>> 
>>> -- ___
>>> Virtuoso-users mailing list
>>> 
>>> Virtuoso-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>>> 
>>> 
>>> 
>> 
>> 
>> 
>> -
>> -
>> 
>> 
>> 
>> ___
>> Virtuoso-users mailing list
>> 
>> Virtuoso-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
> 
> --
>  ___
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users


--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] VirtuosoDataSource fault-tolerance

[Davis, Daniel (NIH/NLM) [C]]

Thanks, Hugh.

We will almost certainly use that now that we have replication. 

 

From: Hugh Williams [mailto:hwilli...@openlinksw.com] 
Sent: Thursday, October 06, 2016 10:09 AM
To: Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov>
Cc: virtuoso-users@lists.sourceforge.net
Subject: Re: [Virtuoso-users] VirtuosoDataSource fault-tolerance

 

Hi Daniel,

 

All the Virtuoso Data Access Clients for JDBC, ODBC, OLEDB, ADO.Net 
<http://ado.net> , Sesame, Jena etc support round robin connections for 
automatic fail-over and load balancing ie HA as detailed at:

 


http://docs.openlinksw.com/virtuoso/dataccessclientsconfailandbalance/

 

As this is in the client it can be used with Virtuoso server instances  of any 
form ie open source , commercial, scale-out cluster etc. provided the nodes are 
known/considered to be in sync and hence can service user requests …

 

Best Regards

Hugh Williams

Professional Services

OpenLink Software, Inc.  //  http://www.openlinksw.com/

Weblog   -- http://www.openlinksw.com/blogs/

LinkedIn -- http://www.linkedin.com/company/openlink-software/

Twitter  -- http://twitter.com/OpenLink

Google+  -- http://plus.google.com/100570109519069333827/

Facebook -- http://www.facebook.com/OpenLinkSoftware

Universal Data Access, Integration, and Management Technology Providers

 

On 4 Oct 2016, at 17:35, Davis, Daniel (NIH/NLM) [C] <daniel.da...@nih.gov 
<mailto:daniel.da...@nih.gov> > wrote:

 

I have a downstream project that uses Virtuoso JDBC against my server.   I am 
wondering whether anyone out there has experience about the fault tolerance of 
VirtuosoDataSource.   

 

That is:

*Does the open source VirtuosoDataSource have any fault tolerance 
features built-in to work with Virtuoso commercial’s replication and clustering 
features?

*Has anyone out there experimented with something like HA-JDBC for 
Virtuoso?

 

Thanks,

 

Dan Davis, Systems/Applications Architect (Contractor),

Office of Computer and Communications Systems,

National Library of Medicine, NIH

 

--
Check out the vibrant tech community on one of the world's most 
engaging tech sites,  <http://slashdot.org/> SlashDot.org!  
<http://sdm.link/slashdot___> 
http://sdm.link/slashdot___
Virtuoso-users mailing list
 <mailto:Virtuoso-users@lists.sourceforge.net> 
Virtuoso-users@lists.sourceforge.net
 <https://lists.sourceforge.net/lists/listinfo/virtuoso-users> 
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

 



smime.p7s
Description: S/MIME cryptographic signature
--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] Restricting DBA logins by client IP

[Davis, Daniel (NIH/NLM) [C]]

I've been hunting over the documentation, and I cannot see the way to restrict 
DBA logins by client IP.   My security team doesn't allow me to use conductor, 
and so I do all my work from isql with the system tables.   Even looking at 
DBEV_LOGIN and FIND_USER hooks, I don't see a way to do this.

Yet, it seems there must be a way.   Can anyone clue me in to what I've missed?

Thanks,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] VirtuosoDataSource fault-tolerance

[Davis, Daniel (NIH/NLM) [C]]

I have a downstream project that uses Virtuoso JDBC against my server.   I am 
wondering whether anyone out there has experience about the fault tolerance of 
VirtuosoDataSource.

That is:

*Does the open source VirtuosoDataSource have any fault tolerance 
features built-in to work with Virtuoso commercial's replication and clustering 
features?

*Has anyone out there experimented with something like HA-JDBC for 
Virtuoso?

Thanks,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Java Melody with Virtuoso

[Davis, Daniel (NIH/NLM) [C]]

So, the problem with this was within my web application, where I cast 
javax.sql.DataSource to virtuoso.jdbc4.VirtuosoDataSource.
With that corrected, JavaMelody is able to monitor the data source.

From: Davis, Daniel (NIH/NLM) [C]
Sent: Tuesday, September 06, 2016 2:33 PM
To: virtuoso-users <virtuoso-users@lists.sourceforge.net>
Subject: [Virtuoso-users] Java Melody with Virtuoso

Has anyone out there done a plugin so that Java Melody can monitor Virtuoso 
JDBC connections?
I have a semantic web application using Virtuoso behind Jena behind Spring, and 
I have Java Melody monitoring it.

It would be great to have Melody collect information on queries, but I'm not 
likely to implement that myself.

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Better question on virt-jena.jar

[Davis, Daniel (NIH/NLM) [C]]

Ah, I see, there's a binsrc/jena, binsrc/jena2, and binsrc/jena3.   I will look 
at the commits to this and try to answer my question on my own.
It would be nice to hear clearly what the commitment of OpenLink is to this.

From: Davis, Daniel (NIH/NLM) [C]
Sent: Monday, November 07, 2016 10:55 AM
To: virtuoso-users@lists.sourceforge.net
Subject: [Virtuoso-users] Better question on virt-jena.jar

Who is maintaining virt-jena?  What is the process of maintaining this, e.g. is 
this portion of virtuoso purely contributed, or does OpenLink plan to update it 
and maintain it.

My application depends on virt-jena.  I will need to tell my boss where an 
upgrade to virt-jena is in an OpenLink roadmap, and also, whether we need to 
dedicate resources ourselves to keep this up to date, or socialize the issue 
with the contributors.  The claimed compatibility of virt-jena.jar is very old, 
and while it works up to 2.13.0, Jena 3 is now over a year old, and has many 
releases.

As a government shop, I ran my webapp dependencies through the OWASP dependency 
checker, and while I could make everything compliant, it is not the best way to 
override individual dependent libraries, and better to get individual dependent 
libraries to be updated.   I'd also love to see the process change to use Maven 
or Ant with ivy.

Thanks,

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] Better question on virt-jena.jar

[Davis, Daniel (NIH/NLM) [C]]

Who is maintaining virt-jena?  What is the process of maintaining this, e.g. is 
this portion of virtuoso purely contributed, or does OpenLink plan to update it 
and maintain it.

My application depends on virt-jena.  I will need to tell my boss where an 
upgrade to virt-jena is in an OpenLink roadmap, and also, whether we need to 
dedicate resources ourselves to keep this up to date, or socialize the issue 
with the contributors.  The claimed compatibility of virt-jena.jar is very old, 
and while it works up to 2.13.0, Jena 3 is now over a year old, and has many 
releases.

As a government shop, I ran my webapp dependencies through the OWASP dependency 
checker, and while I could make everything compliant, it is not the best way to 
override individual dependent libraries, and better to get individual dependent 
libraries to be updated.   I'd also love to see the process change to use Maven 
or Ant with ivy.

Thanks,
Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Better question on virt-jena.jar

[Davis, Daniel (NIH/NLM) [C]]

Idehen/Hugh/Sergei,

I did my own research on this.  OpenLink wrote the initial virt_jena.jar 
driver, and maintains it, but does not enhance it.  Sorry for the dumb 
questions - I can check this stuff on github.com, but sometimes a conversation 
is a good thing.

Sergei, https://github.com/smalinin, github suggests you contributed both the 
Jena2 and Jena3 driver and are still contributing to them.  Looking at the 
commits, OpenLink commits the binary virt_jena2.jar and virt_jena3.jar, and 
changes version numbers, from time to time.

It might be good to communicate about the roadmap for these - I see 
virt_jena3.jar there, but I know my application would need other changes above 
Virtuoso JDBC and virt_jena3 as well to use Jena 3.   If we make these, I will 
think about any suggestions I could make.   Maybe I can make these into 
github.com issues, with or without pull requests depending on my availability.

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH



From: Kingsley Idehen [mailto:kide...@openlinksw.com]
Sent: Monday, November 07, 2016 11:05 AM
To: virtuoso-users@lists.sourceforge.net
Cc: Hugh Williams <hwilli...@openlinksw.com>; Sergei Malinin 
<smali...@openlinksw.com>
Subject: Re: [Virtuoso-users] Better question on virt-jena.jar

On 11/7/16 10:55 AM, Davis, Daniel (NIH/NLM) [C] wrote:
Who is maintaining virt-jena?  What is the process of maintaining this, e.g. is 
this portion of virtuoso purely contributed, or does OpenLink plan to update it 
and maintain it.

My application depends on virt-jena.  I will need to tell my boss where an 
upgrade to virt-jena is in an OpenLink roadmap, and also, whether we need to 
dedicate resources ourselves to keep this up to date, or socialize the issue 
with the contributors.  The claimed compatibility of virt-jena.jar is very old, 
and while it works up to 2.13.0, Jena 3 is now over a year old, and has many 
releases.

As a government shop, I ran my webapp dependencies through the OWASP dependency 
checker, and while I could make everything compliant, it is not the best way to 
override individual dependent libraries, and better to get individual dependent 
libraries to be updated.   I'd also love to see the process change to use Maven 
or Ant with ivy.


Daniel,

We are committed to keeping our native Jena Providers up to date. I'll take a 
look at the Jena 3 issue.

--

Regards,



Kingsley Idehen

Founder & CEO

OpenLink Software   (Home Page: http://www.openlinksw.com)



Weblogs (Blogs):

Legacy Blog: http://www.openlinksw.com/blog/~kidehen/

Blogspot Blog: http://kidehen.blogspot.com

Medium Blog: https://medium.com/@kidehen



Profile Pages:

Pinterest: https://www.pinterest.com/kidehen/

Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen

Twitter: https://twitter.com/kidehen

Google+: https://plus.google.com/+KingsleyIdehen/about

LinkedIn: http://www.linkedin.com/in/kidehen



Web Identities (WebID):

Personal: http://kingsley.idehen.net/dataspace/person/kidehen#this

: 
http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.ttl#this


--
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] Using PREFIX in a prepared Statement

[Davis, Daniel (NIH/NLM) [C]]

So, I must run my application through a source code scanner, and it is 
reasonably complaining that I am using String.format() and 
stmt.executeQuery(query) with the Virtuoso JDBC connection.

Yet, this is SPARQL.   I see some discussion of prepared statements on this 
list in the past.   What I want to know is:

*What is the best way to get this to work at all?

*Is there any way I can bind a parameter whose value includes a PREFIX 
in the query?Something like stmt.setString(1, "mesh:D20189");

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH

--
Check out the vibrant tech community on one of the world's most 
engaging tech sites, SlashDot.org! http://sdm.link/slashdot___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Using PREFIX in a prepared Statement

[Davis, Daniel (NIH/NLM) [C]]

Kingsley,

I'm not sure exactly how this helps me.   Let me be clearer on the problem.   
Because this application runs within a federal government site, I'm required to 
submit my source code to static analysis.   Although the data is read-only, and 
the database user has only read-only access, there are still general guidelines.
I'm inquiring in the sense therefore that I want to figure out the boundaries 
of how Virtuoso JDBC can enable me to pass these guidelines, even though I 
think I can get an exemption by labeling the SQL injection complaint a false 
positive.

What I'm talking about is doing this through a JDBC interface, and rather than 
using a query of the form:

String queryFormat = "SPARQL"
+ " define input:inference \"http://id.nlm.nih.gov/mesh/vocab\";
+ " PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>"
+ " PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>"
+ " PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>"
+ " PREFIX owl: <http://www.w3.org/2002/07/owl#>"
+ " PREFIX meshv: <http://id.nlm.nih.gov/mesh/vocab#>"
+ " PREFIX mesh: <http://id.nlm.nih.gov/mesh/>"
+ " SELECT ?l"
+ " FROM <http://id.nlm.nih.gov/mesh>"
+ " WHERE { mesh:%s %s ?l }";
String query = String.format(queryFormat, id, prop);
log.info(query);
Statement stmt = connection.createStatement();
ResultSet rset = stmt.executeQuery(query);

I would rather do this using a prepared statement, maybe still formatting in 
the property so that the query plan can benefit from the predicate index:

String queryFormat = "SPARQL"
+ " define input:inference \"http://id.nlm.nih.gov/mesh/vocab\";
+ " PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>"
+ " PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>"
+ " PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>"
+ " PREFIX owl: <http://www.w3.org/2002/07/owl#>"
+ " PREFIX meshv: <http://id.nlm.nih.gov/mesh/vocab#>"
+ " PREFIX mesh: <http://id.nlm.nih.gov/mesh/>"
+ " SELECT ?l"
+ " FROM <http://id.nlm.nih.gov/mesh>"
+ " WHERE { ?? %s ?l }";
String query = String.format(queryFormat, prop);
log.info(query);
PreparedStatement stmt = connection.prepareStatement(query);
stmt.setString(1, "http://id.nlm.nih.gov/mesh/"+id);

So far, this general approach of using a prepared statement is not working for 
me.   I may struggle through it, but is there anyway for me to preserve the 
convenience of prefixes:

... WHERE { mesh:?? %s ?l };

And then later bind the query argument?

From: Kingsley Idehen [mailto:kide...@openlinksw.com]
Sent: Friday, October 21, 2016 6:36 PM
To: virtuoso-users@lists.sourceforge.net
Subject: Re: [Virtuoso-users] Using PREFIX in a prepared Statement

On 10/21/16 5:36 PM, Davis, Daniel (NIH/NLM) [C] wrote:
So, I must run my application through a source code scanner, and it is 
reasonably complaining that I am using String.format() and 
stmt.executeQuery(query) with the Virtuoso JDBC connection.

Yet, this is SPARQL.   I see some discussion of prepared statements on this 
list in the past.   What I want to know is:

*What is the best way to get this to work at all?

*Is there any way I can bind a parameter whose value includes a PREFIX 
in the query?Something like stmt.setString(1, "mesh:D20189");

Dan Davis, Systems/Applications Architect (Contractor),
Office of Computer and Communications Systems,
National Library of Medicine, NIH



I assume your query is of the form:

SPARQL

{SPARQL-Query}



OR

SELECT {SELECT-LIST}

FROM {SPARQL {SPARQL-QUERY} } AS {SQL Tabular Relation ALIAS} .



[1] http://bit.ly/sparql-as-sql-tabular-relation -- SPARQL Query as SQL Tabular 
Relation (note: user "vdb" for user and pwd) in FROM CLAUSE

[2] http://bit.ly/sparql-as-sql-tabular-relation2 -- SPARQL Query as SQL 
Tabular Relation

--

Regards,



Kingsley Idehen

Founder & CEO

OpenLink Software   (Home Page: http://www.openlinksw.com)



Weblogs (Blogs):

Legacy Blog: http://www.openlinksw.com/blog/~kidehen/

Blogspot Blog: http://kidehen.blogspot.com

Medium Blog: https://medium.com/@kidehen



Profile Pages:

Pinterest: https://www.pinterest.com/kidehen/

Quora: https://www.quora.com/profile/Kingsley-Uyi-Idehen

Twitter: https://twitter.com/kidehen

Google+: https://plus.google.com/+KingsleyIdehen/about

LinkedIn: http://www.linkedin.com/in/kidehen



Web Identities (WebID):

Pers

Re: [Virtuoso-users] Better question on virt-jena.jar

[Davis, Daniel (NIH/NLM) [C]]

Thanks again.

-Original Message-
From: Sergey Malinin [mailto:sergmali...@gmail.com] 
Sent: Monday, November 07, 2016 11:16 AM
To: Kingsley Idehen <kide...@openlinksw.com>; 
virtuoso-users@lists.sourceforge.net
Cc: Hugh Williams <hwilli...@openlinksw.com>
Subject: Re: [Virtuoso-users] Better question on virt-jena.jar

08/11/16 02:05, Kingsley Idehen wrote:
> On 11/7/16 10:55 AM, Davis, Daniel (NIH/NLM) [C] wrote:
>>
>> Who is maintaining virt-jena?  What is the process of maintaining 
>> this, e.g. is this portion of virtuoso purely contributed, or does OpenLink 
>> plan to update it and maintain it.
>>
>> My application depends on virt-jena.  I will need to tell my boss 
>> where an upgrade to virt-jena is in an OpenLink roadmap, and also, 
>> whether we need to dedicate resources ourselves to keep this up to date, or 
>> socialize the issue with the contributors.  The claimed compatibility of 
>> virt-jena.jar is very old, and while it works up to 2.13.0, Jena 3 is now 
>> over a year old, and has many releases.
>>
>> As a government shop, I ran my webapp dependencies through the OWASP 
>> dependency checker, and while I could make everything compliant, it is not 
>> the best way
>> to override individual dependent libraries, and better to get individual 
>> dependent libraries to be updated.   I'd also love to see the process change 
>> to use
>> Maven or Ant with ivy.
>>
>
> Daniel,
>
> We are committed to keeping our native Jena Providers up to date. I'll take a 
> look at the Jena 3 issue.
>

[Sergey]
For Jena 2.13 the virt_jena2.jar must be used For Jena3  the virt_jena3.jar All 
are in the git open sources.

> --
> Regards,
>
> Kingsley Idehen   
> Founder & CEO
> OpenLink Software   (Home Page:http://www.openlinksw.com)
>
> Weblogs (Blogs):
> Legacy Blog:http://www.openlinksw.com/blog/~kidehen/
> Blogspot Blog:http://kidehen.blogspot.com Medium 
> Blog:https://medium.com/@kidehen
>
> Profile Pages:
> Pinterest:https://www.pinterest.com/kidehen/
> Quora:https://www.quora.com/profile/Kingsley-Uyi-Idehen
> Twitter:https://twitter.com/kidehen
> Google+:https://plus.google.com/+KingsleyIdehen/about
> LinkedIn:http://www.linkedin.com/in/kidehen
>
> Web Identities (WebID):
> Personal:http://kingsley.idehen.net/dataspace/person/kidehen#this
>  
> :http://id.myopenlink.net/DAV/home/KingsleyUyiIdehen/Public/kingsley.t
> tl#this
>


--
Best Regards,
Sergey Malinin

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Virtuoso Jena provider

[Davis, Daniel (NIH/NLM) [C]]

NOTE: there are several editions of virt_jena:

https://github.com/openlink/virtuoso-opensource/blob/develop/7/binsrc/jena/virt_jena.jar
https://github.com/openlink/virtuoso-opensource/blob/develop/7/binsrc/jena2/virt_jena2.jar
https://github.com/openlink/virtuoso-opensource/blob/develop/7/binsrc/jena3/virt_jena3.jar

I wish these and virt_jdbc variants would be published to something like 
Sonatype OSS releases, but aside from that, they work well.

-Original Message-
From: Sergey Malinin [mailto:sergmali...@gmail.com] 
Sent: Monday, June 19, 2017 5:19 AM
To: Henrik Schmidt ; 
virtuoso-users@lists.sourceforge.net
Subject: Re: [Virtuoso-users] Virtuoso Jena provider

On 06/19/2017 06:34 PM, Henrik Schmidt wrote:
> Hi,
>
> is the virtuoso jena provider compatible with jena > 3.0.x like 3.2 or
> 3.3 ? The VOS wiki is only talking abot 3.0.x
>
> Best,
[Sergey]
It works with Jena 3.3.x.
Jena 3.2.x has a bug, that doesn't allows to use it in some cases and create a 
problem with using it with Virtuoso Jena provider.
The bug was fixed in Jena 3.3.x.
>
> Henrik
>
> --
>  Check out the vibrant tech community on one of the world's 
> most engaging tech sites, Slashdot.org! http://sdm.link/slashdot 
> ___
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>

--
Best Regards,
Sergey Malinin



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] how to pass as parameter the virtual graph IRI to a SPARQL statement ?

[Davis, Daniel (NIH/NLM) [C]]

Here’s a brief example:

--
-- Inserts triples of the form ?s meshv:active "true|false"^^xsd:boolean
--
CREATE PROCEDURE meshrdf_insert_active(in graph_uri varchar, in active varchar) 
{
DECLARE stmt varchar;

IF (active <> 'false' AND active <> 'true')
signal('INVALID', 'The  argument must be either "true" or 
"false"');

stmt := sprintf('PREFIX meshv: 
 PREFIX xsd: 
 WITH <%s>
 INSERT { ?s meshv:active "%s"^^xsd:boolean }
 WHERE { ?s a ?o }', graph_uri, active);

LOG_ENABLE(2,1);
CALL SPARQL_EVAL(stmt, graph_uri, 0);
}

When performance is the overriding concern, and when the SQL is sufficiently 
clear, you can use SQL – This is one of the main advantages of using Virtuoso:

--
-- Deletes all triples of the form ?s meshv:active ?o from a particular graph.
--
-- The equivalent SPARUL would be:
--
--  PREFIX meshv: 
--  WITH 
--  DELETE { ?s meshv:active ?a } WHERE { ?s meshv:active ?a }
--
-- SQL is compiled when the stored procedure is defined, even though graph_uri
-- is an argument.   This would not be the case for SPARUL.
--
CREATE PROCEDURE meshrdf_clear_active(in graph_uri varchar) {
LOG_ENABLE(2,1);
DELETE FROM RDF_QUAD
  WHERE g = rdf_make_iid_of_qname(graph_uri)
AND p = 
rdf_make_iid_of_qname('http://id.nlm.nih.gov/mesh/vocab#active');
}

From: Beppe Mazzola [mailto:beppemazz...@gmail.com]
Sent: Monday, June 19, 2017 1:27 PM
To: virtuoso-users@lists.sourceforge.net
Subject: [Virtuoso-users] how to pass as parameter the virtual graph IRI to a 
SPARQL statement ?

Hi to all
I am trying to develop a SPARQL statement in a stored procedure related to a 
virtual graph (result of a R2RML mapping) having its IRI not hard coded, but 
contained in a variable.
I have tried for example the following syntax, but the query does not return 
any triple.


declare graphName varchar;
graphName := 'http://example.com/resource';
for (sparql define input:storage 
"http://www.openlinksw.com/schemas/virtrdf#DefaultQuadStorage;
select distinct ?class where
{
  graph ?:graphName
  {
?s  ?class
  }
} order by ?class
  ) do
{
  -- do something
  string_to_file ('./dump/out.txt',"class" || '\n',-1);
} ;


As soon as I use the hard coded graph name in the SPARQL statement it works 
fine.

...
select distinct ?class where
{
  graph 
  {
?s  ?class
  }
} order by ?class
  ) do
  ...
The same if I switch from a virtual graph to a real one. It works immediately 
with te graph in a variable. Like in the following code.

declare graphName varchar;
graphName := 'http://xmlns.com/foaf/0.1/';
for (sparql define input:storage ""
  select distinct ?class where
   {
 graph ?:graphName
 {
?s  ?class
 }
   } order by ?class limit 10
  ) do
{
  -- do something
  string_to_file ('./dump/out.txt',"class" || '\n',-1);
} ;

Any chance to have a virtual graph IRI contained in a variable in a working 
SPARQL statement?

Thanks!
Cheers
Beppe
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users