Re: [Virtuoso-users] Out of curiosity, some questions about OpenSource Virtuoso settings

2016-09-12 Thread Paul Houle 
The trouble with classic http is that pretty frequently you wind up making
a separate TCP connection for each API call.  In theory there was
pipelining in http/1.1 but it did not really work and was rarely
implemented in "web service" scenarios.

http/2 has pipelining that works,  reduces the # of round trips to do
encryption,  compresses headers,  and has all sorts of goodness.

Another issue is that http-based protocols have historically been
text-based protocols and if there was useful thing I learned in grad school
it was that you can do an awful lot of FLOPS in the time that it takes to
parse a string like "7.55523135E12".

Prior to 2000,  it was common to see a wide range of text-based formats for
structured data such as HL7,  UN/EDIFACT,  the communication packets used
in amateur radio, etc.

In the 2000-2010 period,  XML ate most of that.  In 2010-present,  JSON has
replaced XML in a lot of places.  As people catch on to what JSON-LD really
means (you can paint on the XML semantics that are missing in JSON with a
minimum of bother),  JSON will grow.

Performance-wise you can get much better results with binary formats and
people today are even realizing (slower than with JSON-LD) you can
implement binary formats in LD in a secure way in C,  unlike text formats.

The interesting thing though is that "general" binary formats are nowhere
near the consolidation we've seen with XML and JSON.  Binary XML formats
have been tried and not caught on outside of niches.  A small community
speaks Binary FIX.  There is protocol buffers,  Captain Proto,  Thrift,
 Avro, MessagePack,  and who knows what else.

Binary formats might never get their XML moment because once you start
caring about performance you care about performance for *your specific
case* and thus it is impossible to pick one protocol that will make
everybody equally happy (or miserable)

Caching is another http family problem.  Once you start caching you run
into (a) the risk that cached data will be invalid,  and (b) there is no
guarantee that perceived performance will be better when you use a cache.
It's is very hard for Homo Corporatus to understand that users (ex.
customers,  employees) experience latency,  not throughput.  (It's anathema
to corporate ideology,  for instance,  that 9 women can't make a baby in 1
month -- I knew Sun was on the skids when they started talking about
"Throughput Computing")

A 5400 rpm hard drive typical in a laptop can spin around about twice in
the time it takes my (awful) DSL connection to round trip to the Azure data
center in Chicago,  and any kind of I/O storm means it can take much longer
than two spins to discover that something is not in the cache.

It's certainly possible that http caching can improve perceived
performance,  but you can't take it for granted.

On Mon, Sep 12, 2016 at 10:49 AM, Davis, Daniel (NIH/NLM) [C] <
daniel.da...@nih.gov> wrote:

> Tangentially on JDBC/ODBC vs HTTP.HTTP is slow because of the
> transport, but JDBC/ODBC has more mature models of paging over results,
> pre-fetching, etc.   I feel that RDF is mature as a Data format, but as a
> protocol it is not that mature.
>
> -Original Message-
> From: Hugh Williams [mailto:hwilli...@openlinksw.com]
> Sent: Monday, September 12, 2016 5:30 AM
> To: Lorenz Buehmann <buehm...@informatik.uni-leipzig.de>
> Cc: virtuoso-users@lists.sourceforge.net
> Subject: Re: [Virtuoso-users] Out of curiosity, some questions about
> OpenSource Virtuoso settings
>
> Hi Lorenz,
>
> In Virtuoso 7+  Vectored query execution [1] enables single, typically
> complex analytical types queries, to be broken down and executed on
> multiple threads. The INI file param that controls this is
> "ThreadsPerQuery" [2] , which controls the maximum number of threads that
> can be claimed from the thread pool by a single query, and there are other
> associated params details at [2] .
>
> [1] http://docs.openlinksw.com/virtuoso/vexqrparl.html
> [2] http://docs.openlinksw.com/virtuoso/vexqrparlconfp/
>
> Best Regards
> Hugh Williams
> Professional Services
> OpenLink Software, Inc.  //  http://www.openlinksw.com/
> Weblog   -- http://www.openlinksw.com/blogs/
> LinkedIn -- http://www.linkedin.com/company/openlink-software/
> Twitter  -- http://twitter.com/OpenLink
> Google+  -- http://plus.google.com/100570109519069333827/
> Facebook -- http://www.facebook.com/OpenLinkSoftware
> Universal Data Access, Integration, and Management Technology Providers
>
> > On 12 Sep 2016, at 07:13, Lorenz Buehmann <buehm...@informatik.uni-
> leipzig.de> wrote:
> >
> > Hi,
> >
> > just as a follow-up question:
> >
> > I know it supports inter-query parallelization, but does it also support
> intra-query parallelization, i.e. using multip

Re: [Virtuoso-users] Out of curiosity, some questions about OpenSource Virtuoso settings

2016-09-12 Thread Davis, Daniel (NIH/NLM) [C] 
Tangentially on JDBC/ODBC vs HTTP.HTTP is slow because of the transport, 
but JDBC/ODBC has more mature models of paging over results, pre-fetching, etc. 
  I feel that RDF is mature as a Data format, but as a protocol it is not that 
mature.

-Original Message-
From: Hugh Williams [mailto:hwilli...@openlinksw.com] 
Sent: Monday, September 12, 2016 5:30 AM
To: Lorenz Buehmann <buehm...@informatik.uni-leipzig.de>
Cc: virtuoso-users@lists.sourceforge.net
Subject: Re: [Virtuoso-users] Out of curiosity, some questions about OpenSource 
Virtuoso settings

Hi Lorenz,

In Virtuoso 7+  Vectored query execution [1] enables single, typically complex 
analytical types queries, to be broken down and executed on multiple threads. 
The INI file param that controls this is "ThreadsPerQuery" [2] , which controls 
the maximum number of threads that can be claimed from the thread pool by a 
single query, and there are other associated params details at [2] .

[1] http://docs.openlinksw.com/virtuoso/vexqrparl.html
[2] http://docs.openlinksw.com/virtuoso/vexqrparlconfp/

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers

> On 12 Sep 2016, at 07:13, Lorenz Buehmann 
> <buehm...@informatik.uni-leipzig.de> wrote:
> 
> Hi,
> 
> just as a follow-up question:
> 
> I know it supports inter-query parallelization, but does it also support 
> intra-query parallelization, i.e. using multiple threads to compute the 
> result of a single query? If yes, which parameter is used to configure this?
> 
> 
> Cheers,
> 
> Lorenz
> 
> On 12.09.2016 05:57, Kingsley Idehen wrote:
>> On 9/10/16 10:41 PM, giacom...@libero.it
>>  wrote:
>> 
>>> So, I have a bunch of questions about Virtuoso.
>>> 
>>> * Are SPARQL queries performed concurrently (using the standard 
>>> virtuoso.ini configuration)?
>>> 
>> Yes, Virtuoso is highly multi-threaded. In addition, it has 
>> vectorized query execution i.e., many query batches per thread, handled 
>> concurrently .
>> 
>>> * Are Virtuoso transactions autocommittable, or there aren't any 
>>> kind of transactions when connecting to the ODBC driver?
>>> 
>> Depends, by default you have Read Committed Isolation level.
>> 
>>> * Does Virtuoso automatically performs triple indexing before query data? 
>>> 
>> No, it has indexes in place.
>> 
>> 
>>> In
>>> some systems I have to configure it manually.
>>> * Does the usage of ODBC degradates the benchmarking of Virtuoso's 
>>> SPARQL query or the time required to store the triples within a given named 
>>> graph?
>>> 
>> No, if anything that's faster than HTTP.
>> 
>> Links:
>> 
>> [1]
>> http://docs.openlinksw.com/virtuoso/isolation/
>> 
>> [2]
>> 
>> http://wikis.openlinksw.com/VirtuosoWikiWeb/ChangeVirtuosoSDefaultTra
>> nsactionIsolationLevel
>> 
>> [3]
>> http://docs.openlinksw.com/virtuoso/fn_log_enable.html
>> 
>> [4]
>> 
>> http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFPer
>> formanceTuning
>> 
>> 
>> [5]
>> 
>> http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtBulkRD
>> FLoader#Running
>> 
>> multiple Loaders
>> 
>> 
>> Kingsley
>> 
>> 
>>> Thanks in advance for any support,
>>> 
>>>> JackB
>>>> 
>>> 
>>> -- ___
>>> Virtuoso-users mailing list
>>> 
>>> Virtuoso-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>>> 
>>> 
>>> 
>> 
>> 
>> 
>> -
>> -
>> 
>> 
>> 
>> ___
>> Virtuoso-users mailing list
>> 
>> Virtuoso-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
> 
> --
>  ___
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users


--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Out of curiosity, some questions about OpenSource Virtuoso settings

2016-09-12 Thread Hugh Williams 
Hi Lorenz,

In Virtuoso 7+  Vectored query execution [1] enables single, typically complex 
analytical types queries, to be broken down and executed on multiple threads. 
The INI file param that controls this is “ThreadsPerQuery” [2] , which controls 
the maximum number of threads that can be claimed from the thread pool by a 
single query, and there are other associated params details at [2] .

[1] http://docs.openlinksw.com/virtuoso/vexqrparl.html
[2] http://docs.openlinksw.com/virtuoso/vexqrparlconfp/

Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc.  //  http://www.openlinksw.com/
Weblog   -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter  -- http://twitter.com/OpenLink
Google+  -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers

> On 12 Sep 2016, at 07:13, Lorenz Buehmann 
>  wrote:
> 
> Hi,
> 
> just as a follow-up question:
> 
> I know it supports inter-query parallelization, but does it also support 
> intra-query parallelization, i.e. using multiple threads to compute the 
> result of a single query? If yes, which parameter is used to configure this?
> 
> 
> Cheers,
> 
> Lorenz
> 
> On 12.09.2016 05:57, Kingsley Idehen wrote:
>> On 9/10/16 10:41 PM, giacom...@libero.it
>>  wrote:
>> 
>>> So, I have a bunch of questions about Virtuoso.
>>> 
>>> * Are SPARQL queries performed concurrently (using the standard 
>>> virtuoso.ini 
>>> configuration)?
>>> 
>> Yes, Virtuoso is highly multi-threaded. In addition, it has vectorized
>> query execution i.e., many query batches per thread, handled concurrently .
>> 
>>> * Are Virtuoso transactions autocommittable, or there aren't any kind of 
>>> transactions when connecting to the ODBC driver?
>>> 
>> Depends, by default you have Read Committed Isolation level.
>> 
>>> * Does Virtuoso automatically performs triple indexing before query data? 
>>> 
>> No, it has indexes in place.
>> 
>> 
>>> In 
>>> some systems I have to configure it manually.
>>> * Does the usage of ODBC degradates the benchmarking of Virtuoso's SPARQL 
>>> query or the time required to store the triples within a given named graph?
>>> 
>> No, if anything that's faster than HTTP.
>> 
>> Links:
>> 
>> [1] 
>> http://docs.openlinksw.com/virtuoso/isolation/
>> 
>> [2]
>> 
>> http://wikis.openlinksw.com/VirtuosoWikiWeb/ChangeVirtuosoSDefaultTransactionIsolationLevel
>> 
>> [3] 
>> http://docs.openlinksw.com/virtuoso/fn_log_enable.html
>> 
>> [4]
>> 
>> http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFPerformanceTuning
>> 
>> 
>> [5]
>> 
>> http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtBulkRDFLoader#Running
>> 
>> multiple Loaders
>> 
>> 
>> Kingsley
>> 
>> 
>>> Thanks in advance for any support,
>>> 
 JackB
 
>>> --
>>> ___
>>> Virtuoso-users mailing list
>>> 
>>> Virtuoso-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>>> 
>>> 
>>> 
>> 
>> 
>> 
>> --
>> 
>> 
>> 
>> ___
>> Virtuoso-users mailing list
>> 
>> Virtuoso-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
> 
> --
> ___
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users



smime.p7s
Description: S/MIME cryptographic signature
--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Out of curiosity, some questions about OpenSource Virtuoso settings

2016-09-12 Thread Lorenz Buehmann 
Hi,

just as a follow-up question:

I know it supports inter-query parallelization, but does it also support
intra-query parallelization, i.e. using multiple threads to compute the
result of a single query? If yes, which parameter is used to configure this?


Cheers,

Lorenz


On 12.09.2016 05:57, Kingsley Idehen wrote:
> On 9/10/16 10:41 PM, giacom...@libero.it wrote:
>> So, I have a bunch of questions about Virtuoso.
>>
>> * Are SPARQL queries performed concurrently (using the standard virtuoso.ini 
>> configuration)?
> Yes, Virtuoso is highly multi-threaded. In addition, it has vectorized
> query execution i.e., many query batches per thread, handled concurrently .
>> * Are Virtuoso transactions autocommittable, or there aren't any kind of 
>> transactions when connecting to the ODBC driver?
> Depends, by default you have Read Committed Isolation level.
>> * Does Virtuoso automatically performs triple indexing before query data? 
> No, it has indexes in place.
>
>> In 
>> some systems I have to configure it manually.
>> * Does the usage of ODBC degradates the benchmarking of Virtuoso's SPARQL 
>> query or the time required to store the triples within a given named graph?
> No, if anything that's faster than HTTP.
>
> Links:
>
> [1] http://docs.openlinksw.com/virtuoso/isolation/
> [2]
> http://wikis.openlinksw.com/VirtuosoWikiWeb/ChangeVirtuosoSDefaultTransactionIsolationLevel
> [3] http://docs.openlinksw.com/virtuoso/fn_log_enable.html
> [4]
> http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFPerformanceTuning
>
> [5]
> http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtBulkRDFLoader#Running
> multiple Loaders
>
>
> Kingsley
>
>> Thanks in advance for any support,
>>> JackB
>> --
>> ___
>> Virtuoso-users mailing list
>> Virtuoso-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>>
>
>
>
> --
>
>
> ___
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Out of curiosity, some questions about OpenSource Virtuoso settings

2016-09-11 Thread Kingsley Idehen 
On 9/10/16 10:41 PM, giacom...@libero.it wrote:
> So, I have a bunch of questions about Virtuoso.
>
> * Are SPARQL queries performed concurrently (using the standard virtuoso.ini 
> configuration)?
Yes, Virtuoso is highly multi-threaded. In addition, it has vectorized
query execution i.e., many query batches per thread, handled concurrently .
> * Are Virtuoso transactions autocommittable, or there aren't any kind of 
> transactions when connecting to the ODBC driver?

Depends, by default you have Read Committed Isolation level.
> * Does Virtuoso automatically performs triple indexing before query data? 
No, it has indexes in place.

> In 
> some systems I have to configure it manually.
> * Does the usage of ODBC degradates the benchmarking of Virtuoso's SPARQL 
> query or the time required to store the triples within a given named graph?

No, if anything that's faster than HTTP.

Links:

[1] http://docs.openlinksw.com/virtuoso/isolation/
[2]
http://wikis.openlinksw.com/VirtuosoWikiWeb/ChangeVirtuosoSDefaultTransactionIsolationLevel
[3] http://docs.openlinksw.com/virtuoso/fn_log_enable.html
[4]
http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFPerformanceTuning

[5]
http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtBulkRDFLoader#Running
multiple Loaders


Kingsley

>
> Thanks in advance for any support,
>> JackB
> --
> ___
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>


-- 
Regards,

Kingsley Idehen   
Founder & CEO 
OpenLink Software   (Home Page: http://www.openlinksw.com)

Medium Blog: https://medium.com/@kidehen
Blogspot Blog: http://kidehen.blogspot.com
Twitter Profile: https://twitter.com/kidehen
Google+ Profile: https://plus.google.com/+KingsleyIdehen/about
LinkedIn Profile: http://www.linkedin.com/in/kidehen
Personal WebID: http://kingsley.idehen.net/dataspace/person/kidehen#this




smime.p7s
Description: S/MIME Cryptographic Signature
--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

Re: [Virtuoso-users] Out of curiosity, some questions about OpenSource Virtuoso settings

2016-09-11 Thread Hugh Williams 
Hi,

> On 11 Sep 2016, at 03:41, giacom...@libero.it wrote:
> 
> So, I have a bunch of questions about Virtuoso.
> 
> * Are SPARQL queries performed concurrently (using the standard virtuoso.ini 
> configuration)?

[Hugh] Yes, although you shoud tune your Virtuoso server for the machine type 
it is running on as detailed in the performance tuning guide:


http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtRDFPerformanceTuning
 


> * Are Virtuoso transactions autocommittable, or there aren't any kind of 
> transactions when connecting to the ODBC driver?

[Hugh] Yes, Virtuoso support all database transaction isolation which can also 
be applied in SPARQL/RDF much as with SQL:

http://docs.openlinksw.com/virtuoso/ptune.html 


> * Does Virtuoso automatically performs triple indexing before query data? In 
> some systems I have to configure it manually.

[Hugh] A default set of indexes are create for the RDF Quad Store when the 
database is created as indicated in the performance tuning guides and data 
loaded is automatically Indexed, thus nothing to manually create …

> * Does the usage of ODBC degradates the benchmarking of Virtuoso's SPARQL 
> query or the time required to store the triples within a given named graph?

[Hugh] No, there will be no performance degradation performed SPARQL operation 
via ODBC or any other SQL ( ie JDBC, ADO.Net etc ) interface.

Best Regards
Hugh Williams
OpenLink Software

> 
> Thanks in advance for any support,
>> JackB
> 
> --
> ___
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users



smime.p7s
Description: S/MIME cryptographic signature
--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users

[Virtuoso-users] Out of curiosity, some questions about OpenSource Virtuoso settings

2016-09-10 Thread giacom...@libero.it 
So, I have a bunch of questions about Virtuoso.

* Are SPARQL queries performed concurrently (using the standard virtuoso.ini 
configuration)?
* Are Virtuoso transactions autocommittable, or there aren't any kind of 
transactions when connecting to the ODBC driver?
* Does Virtuoso automatically performs triple indexing before query data? In 
some systems I have to configure it manually.
* Does the usage of ODBC degradates the benchmarking of Virtuoso's SPARQL 
query or the time required to store the triples within a given named graph?

Thanks in advance for any support,
> JackB

--
___
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users