Re: Fuseki service vs. webapp

[Andy Seaborne]

If the client software is not closing connections properly (and due to 
streaming, Content-Length isn't set so the client must call "close"), 
then there is a build-up of open sockets.


This is especially true if the client and server are on the same 
machine. Even then, it is possible to over whelm the operation system.


Both those Fuseki forms (standalone, and war file) are running as 
webapps. The standalone server is using a version Jetty as the container.


Embedded is another matter - it used Jetty and again it is possible to 
flood it if the OS does not get a chance clear up TCP connections fast 
enough. This is asynchronous so if the local machine isn't handling some 
CPU time to the kernel, they hang around for a long time (2 minutes).


Andy

On 26/04/18 13:54, Rob Vesse wrote:

Sorin

Thanks for the report, unfortunately the lack of reproducible detail means I 
can only speculate on the source of the issues.

The Jena project has only recently added a template service file for Fuseki so 
presumably when you say you are setting it up as a service either you define 
your own service definition or you use our template as a starting point?  If 
you use your own it would be useful to see that.

I believe that the service template we provide invokes the embedded Fuseki 
server JAR which is embedded Jetty as its webapp server, you can optionally 
customise the Jetty config (and thus connection handling) by providing the 
--jetty-config option with an appropriate config file.  If no such config is 
provided Fuseki will set up a basic default config (see 
https://github.com/apache/jena/blob/master/jena-fuseki2/jena-fuseki-core/src/main/java/org/apache/jena/fuseki/jetty/JettyFuseki.java#L304)
 which presumably is insufficient for your use case.  However since you haven't 
specified any details I can only assume you are using embedded Fuseki and this 
could be some other problem entirely.

When run as a webapp then connection handling will be defined by the webapp 
server you choose to use and its configuration.  Again you haven't specified 
what webapp server you use nor provided any detail about its configuration.  So 
we have no idea what the connection handling setup is relative to the basic 
default config in the embedded case.

In summary:
- Embedded Fuseki provides a default setup that is sufficient for basic users 
but can be customised if desired
- Webapp Fuseki allows you to run it in the webapp server of your choice 
customised as desired

If you have suggestions for how we might better define the default 
configuration for the embedded case then contributions in that regard would be 
welcome.  Alternatively if the guesses I have made here doesn't apply to your 
actual environment then providing a complete description of how to reproduce 
the problem would be useful.

Thanks,

Rob

On 26/04/2018, 12:56, "Sorin Gheorghiu"  
wrote:

 Hi,
 
 Let's suppose Fuseki 3.6.0 is started as a service (and specific plugins

 like reindex, percolator, mustache, netty3 & 4 are loaded). Then a
 script which reads intensively data over SPARQL fails after the first
 ~400 SELECT's with connection refused error
 (Net::HTTP::Persistent::Error). It seems due to too many client
 connections in use fuseki is not capable to open new ones.
 
 If the log4j is set to DEBUG (instead INFO or WARN) the script won't

 crash, but fuseki will generate a huge amount of debug data which
 produces a process delay which helps it to close the old connections. Of
 course, this is not a reliable workaround on a production server.
 
 Alternatively, if fuseki is started as a Webapp (for the same assembler

 file), then the same script runs (more than 8 SELECT's within 1
 hour) without failure. In conclusion, fuseki as a Web Application
 handles better the client connections.
 
 Regards,

 Sorin
 
 
 

Re: Fuseki service vs. webapp

[Jean-Marc Vanel]

Hi Sorin,

you might want to try semantic_forms,
another SPARQL server based on Jena TDB.
This one is developed in Scala, on top of PlayFramework,
itself on top of Netty 4.0.41.Final .
Semantic_forms is also a generic semantic web navigation and annotation
application, and a framework to build business applications centered on
forms (input forms or read only).

A pre-release is here :
https://github.com/jmvanel/semantic_forms/releases/tag/2.5.0

You can test the zip,
and/or the sites:
http://semantic-forms.cc:9112 : general site
http://semantic-forms.cc:9111 : sandbox

*NOTES:*

   - *currently , a Jena assembly file is not taken in account, but I can
   add that quickly if you ask*
   - *instead, a single TDB directory , named "TDB" is  taken in account*




2018-04-26 13:55 GMT+02:00 Sorin Gheorghiu 
:

> Hi,
>
> Let's suppose Fuseki 3.6.0 is started as a service (and specific plugins
> like reindex, percolator, mustache, netty3 & 4 are loaded). Then a script
> which reads intensively data over SPARQL fails after the first ~400
> SELECT's with connection refused error (Net::HTTP::Persistent::Error). It
> seems due to too many client connections in use fuseki is not capable to
> open new ones.
>
> If the log4j is set to DEBUG (instead INFO or WARN) the script won't
> crash, but fuseki will generate a huge amount of debug data which produces
> a process delay which helps it to close the old connections. Of course,
> this is not a reliable workaround on a production server.
>
> Alternatively, if fuseki is started as a Webapp (for the same assembler
> file), then the same script runs (more than 8 SELECT's within 1 hour)
> without failure. In conclusion, fuseki as a Web Application handles better
> the client connections.
>
> Regards,
> Sorin
>
>
>


-- 
Jean-Marc Vanel
http://www.semantic-forms.cc:9111/display?displayuri=http://jmvanel.free.fr/jmv.rdf%23me#subject

Déductions SARL - Consulting, services, training,
Rule-based programming, Semantic Web
+33 (0)6 89 16 29 52
Twitter: @jmvanel , @jmvanel_fr ; chat: irc://irc.freenode.net#eulergui

Re: Fuseki service vs. webapp

[Rob Vesse]

Sorin

Thanks for the report, unfortunately the lack of reproducible detail means I 
can only speculate on the source of the issues.

The Jena project has only recently added a template service file for Fuseki so 
presumably when you say you are setting it up as a service either you define 
your own service definition or you use our template as a starting point?  If 
you use your own it would be useful to see that.

I believe that the service template we provide invokes the embedded Fuseki 
server JAR which is embedded Jetty as its webapp server, you can optionally 
customise the Jetty config (and thus connection handling) by providing the 
--jetty-config option with an appropriate config file.  If no such config is 
provided Fuseki will set up a basic default config (see 
https://github.com/apache/jena/blob/master/jena-fuseki2/jena-fuseki-core/src/main/java/org/apache/jena/fuseki/jetty/JettyFuseki.java#L304)
 which presumably is insufficient for your use case.  However since you haven't 
specified any details I can only assume you are using embedded Fuseki and this 
could be some other problem entirely.

When run as a webapp then connection handling will be defined by the webapp 
server you choose to use and its configuration.  Again you haven't specified 
what webapp server you use nor provided any detail about its configuration.  So 
we have no idea what the connection handling setup is relative to the basic 
default config in the embedded case.

In summary:
- Embedded Fuseki provides a default setup that is sufficient for basic users 
but can be customised if desired
- Webapp Fuseki allows you to run it in the webapp server of your choice 
customised as desired

If you have suggestions for how we might better define the default 
configuration for the embedded case then contributions in that regard would be 
welcome.  Alternatively if the guesses I have made here doesn't apply to your 
actual environment then providing a complete description of how to reproduce 
the problem would be useful.

Thanks,

Rob

On 26/04/2018, 12:56, "Sorin Gheorghiu"  
wrote:

Hi,

Let's suppose Fuseki 3.6.0 is started as a service (and specific plugins 
like reindex, percolator, mustache, netty3 & 4 are loaded). Then a 
script which reads intensively data over SPARQL fails after the first 
~400 SELECT's with connection refused error 
(Net::HTTP::Persistent::Error). It seems due to too many client 
connections in use fuseki is not capable to open new ones.

If the log4j is set to DEBUG (instead INFO or WARN) the script won't 
crash, but fuseki will generate a huge amount of debug data which 
produces a process delay which helps it to close the old connections. Of 
course, this is not a reliable workaround on a production server.

Alternatively, if fuseki is started as a Webapp (for the same assembler 
file), then the same script runs (more than 8 SELECT's within 1 
hour) without failure. In conclusion, fuseki as a Web Application 
handles better the client connections.

Regards,
Sorin

Fuseki service vs. webapp

[Sorin Gheorghiu]

Hi,

Let's suppose Fuseki 3.6.0 is started as a service (and specific plugins 
like reindex, percolator, mustache, netty3 & 4 are loaded). Then a 
script which reads intensively data over SPARQL fails after the first 
~400 SELECT's with connection refused error 
(Net::HTTP::Persistent::Error). It seems due to too many client 
connections in use fuseki is not capable to open new ones.


If the log4j is set to DEBUG (instead INFO or WARN) the script won't 
crash, but fuseki will generate a huge amount of debug data which 
produces a process delay which helps it to close the old connections. Of 
course, this is not a reliable workaround on a production server.


Alternatively, if fuseki is started as a Webapp (for the same assembler 
file), then the same script runs (more than 8 SELECT's within 1 
hour) without failure. In conclusion, fuseki as a Web Application 
handles better the client connections.


Regards,
Sorin