Re: Fuseki 2+- Lucene doesn't play well with Tomcat Manager

Fri, 25 Sep 2015 11:45:12 -0700

Thanks again Andy. I upgraded to Fuseki 2.3.0 / Jena 3.0.0, no difference in how it behaves with Tomcat start/stop service. You asked what I'd expect to happen with active transactions. If they are transactions, I'd expect them to complete or be undone. Really, I just want the same behavior that occurs with starting and stopping the complete running Tomcat instance (which works, but it forces me to bring down other services).
By large files (with regard to SPARQL upload vs. bulkloader) I mean large for an ontology, not large for data. I'm talking about 2 files, each with about 2.x million triples. 


So, I'm now trying to just use sparql update, but I can't get it to 
work. I don't get any error, it just hangs indefinitely on a command like:


s-update --service http://localhost:8080/fuseki/sig/update 'CLEAR DEFAULT'


I'm thinking that it may be in read-only mode somehow? I've seen docs 
mentioning passing the argument --update to the fuseki server. But I'm 
not sure how this works if launched via Tomcat. Any suggestions that you 
might have here would be much appreciated.


Thanks,
Todd

Landon Todd Detwiler
Structural Informatics Group (SIG)
University of Washington

phone: 206-351-7721

On 9/16/15 12:44 PM, Andy Seaborne wrote:

On 15/09/15 21:00, Todd Detwiler wrote:

Hi Andy,
In this case I am running the version of Jena that came with a recent
(within the last couple of months) Fuseki2 war file distribution. The
core libs look to be 2.13.*.


Fuseki 2.3.0 / Jena 3.0.0 was July.



I can't say what Tomcat does under the hood when you start and stop
apps, but the behavior I saw wasn't just through the text-based http
tools, the same behavior occurs via the manager gui. I know that there
are start-up and shutdown listener hooks in Tomcat. So I could insert
some clean-up code there if I knew what needed to happen.



I don't know about the text index.  For TDB, something needs to 
release the database (StoreConnection.reset() - caution - what about 
active transactions?)



Yes, I could do as you say and limit the update endpoint to localhost.
Will rebuilding the whole store (I want to drop everything in it,
rebuild from two large files, and regenerate the Lucene indexes) be as
efficient as using the bulk loader? Is that your recommended method?



It depends on "large".  The TDB bulkloader is faster than just loading 
via the usual API.  Maybe the same for the text index.


    Andy



Your last sentence said,


SPARQL Update (in the latest version where text changes and database
changes are coordinated) is


What did you intend to say after "is"?

Thanks,
Todd

Landon Todd Detwiler
Structural Informatics Group (SIG)
University of Washington

phone: 206-351-7721

On 9/15/15 12:46 PM, Andy Seaborne wrote:

On 15/09/15 17:39, Todd Detwiler wrote:

Let me start with what I'm trying to do. Basically, I'm writing a

script, that will run off of a cron job, that will rebuild my TDB 
store

(and Lucene indexes) on a regular basis. I looked into using the TDB

command line scripts, but they fail because another JVM (the tomcat 
app)

is already using the TDB store. I looked into using Sparql update
through Fuseki, but I like the idea of the store being read only (and

therefore update is not allowed). So, I am experimenting with 
having my
script interact with the Tomcat Manager, stopping the running 
Fuseki app
(but not all of tomcat), making changes, and then restarting. I am 
able

to stop it fine, using a url like:

(domain):(port)/manager/text/stop?path=/fuseki

However, the corresponding call to start fails:

(domain):(port)/manager/text/start?path=/fuseki

I am getting a stack trace in the tomcat log file that indicates an

issue with file locks related to Lucene indexing. Now, if I stop 
tomcat
entirely, and start it again, everything works fine. But I don't 
want to
shut down the other services. Do I need to add a listener that 
performs

some sort of operation when Tomcat stops? Have others encountered the

same issue. I've included the stack trace at the bottom of this 
message.

I'd appreciate any insight anyone could offer. Or let me know if you
think using Sparql update would be a preferred way of rebuilding the
store?

Thanks,
Todd


Hi Todd,

Firstly, which version of jena are you using? There were some changes
to the Lucene indexing around updates that changed the way it works
which may play into this.  I see "com.hp.hpl.jena" so it's not the
latest.

Also, I don't know what stopping a webapp in Tomcat actually does. The
standard ServletContextListener only has an operation called
synchronously so it is tricky to clear up in-flight transactions (TDB
does not have forced system aborts by design). Fuseki2, and maybe
wider, needs a proper service lifecycle.

Restarting is clearly calling assembler processing.

Could you protect the update endpoints with the tomcat configuration
to, say, only respond to requests for localhost? While forcing
listening to a specific host isn't very safe, localhost is different
because it means the request was on the loop back interface so can't
be done from outside.

SPARQL Update (in the latest version where text changes and database
changes are coordinated) is

    Andy



15-Sep-2015 09:02:55.968 SEVERE [http-nio-8080-exec-10]

org.apache.catalina.core.StandardContext.listenerStart Exception 
sending

context initialized event to listener instance of class
org.apache.jena.fuseki.server.FusekiServerListener

  com.hp.hpl.jena.assembler.exceptions.AssemblerException: caught: 
null

   doing:
     root: http://localhost/jena_example/#indexLucene with type:
http://jena.apache.org/text#TextIndexLucene assembler class: class
org.apache.jena.query.text.assembler.TextIndexLuceneAssembler
     root: http://localhost/jena_example/#text_dataset with type:
http://jena.apache.org/text#TextDataset assembler class: class
org.apache.jena.query.text.assembler.TextDatasetAssembler

     at

com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.openBySpecificType(AssemblerGroup.java:138) 




     at

com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.open(AssemblerGroup.java:117) 




     at

com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$ExpandingAssemblerGroup.open(AssemblerGroup.java:81) 




     at

com.hp.hpl.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:52) 




     at

com.hp.hpl.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:48) 




     at

org.apache.jena.query.text.assembler.TextDatasetAssembler.open(TextDatasetAssembler.java:63) 




     at

org.apache.jena.query.text.assembler.TextDatasetAssembler.open(TextDatasetAssembler.java:41) 




     at

com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.openBySpecificType(AssemblerGroup.java:130) 




     at

com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.open(AssemblerGroup.java:117) 




     at

com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$ExpandingAssemblerGroup.open(AssemblerGroup.java:81) 




     at

com.hp.hpl.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:52) 




     at

com.hp.hpl.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:48) 




     at
org.apache.jena.fuseki.build.Builder.buildDataService(Builder.java:76)
     at

org.apache.jena.fuseki.build.Builder.buildDataAccessPoint(Builder.java:61) 



     at

org.apache.jena.fuseki.build.FusekiConfig.servicesAndDatasets(FusekiConfig.java:133) 




     at

org.apache.jena.fuseki.build.FusekiConfig.readConfigFile(FusekiConfig.java:75) 




     at

org.apache.jena.fuseki.server.FusekiServer.initServerConfiguration(FusekiServer.java:248) 




     at

org.apache.jena.fuseki.server.FusekiServer.initializeDataAccessPoints(FusekiServer.java:216) 




     at

org.apache.jena.fuseki.server.FusekiServerListener.init(FusekiServerListener.java:70) 




     at

org.apache.jena.fuseki.server.FusekiServerListener.contextInitialized(FusekiServerListener.java:45) 




     at

org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4729) 




     at

org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5167) 




     at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
     at

org.apache.catalina.manager.ManagerServlet.start(ManagerServlet.java:1293) 



     at

org.apache.catalina.manager.HTMLManagerServlet.start(HTMLManagerServlet.java:673) 




     at

org.apache.catalina.manager.HTMLManagerServlet.doPost(HTMLManagerServlet.java:221) 




     at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
     at

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) 




     at

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) 




     at

org.apache.catalina.filters.CsrfPreventionFilter.doFilter(CsrfPreventionFilter.java:213) 




     at

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) 




     at

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) 




     at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
     at

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) 




     at

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) 




     at

org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108) 




     at

org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) 




     at

org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) 




     at

org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219) 




     at

org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) 




     at

org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:614) 




     at

org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142) 




     at

org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) 




     at

org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617) 




     at

org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) 




     at

org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518) 



     at

org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091) 




     at

org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668) 




     at

org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1521) 




     at

org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1478) 




     at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 




     at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 




     at

org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) 




     at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.OverlappingFileLockException

     at 
sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)

     at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
     at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1075)
     at java.nio.channels.FileChannel.tryLock(FileChannel.java:1155)
     at

org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:217) 



     at org.apache.lucene.store.Lock.obtain(Lock.java:72)

     at 
org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:702)

     at

org.apache.jena.query.text.TextIndexLucene.openIndexWriter(TextIndexLucene.java:121) 




     at

org.apache.jena.query.text.TextIndexLucene.<init>(TextIndexLucene.java:114) 



     at

org.apache.jena.query.text.TextDatasetFactory.createLuceneIndex(TextDatasetFactory.java:97) 




     at

org.apache.jena.query.text.assembler.TextIndexLuceneAssembler.open(TextIndexLuceneAssembler.java:96) 




     at

org.apache.jena.query.text.assembler.TextIndexLuceneAssembler.open(TextIndexLuceneAssembler.java:47) 




     at

com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.openBySpecificType(AssemblerGroup.java:130) 




     ... 54 more































					Reply via email to