Hi Todd,
Sorry for the delay. Minor tidal wave of email.
On 25/09/15 19:41, Todd Detwiler wrote:
Thanks again Andy. I upgraded to Fuseki 2.3.0 / Jena 3.0.0, no
difference in how it behaves with Tomcat start/stop service.
Useful information. I created JENA-1043 to record this overall.
There is now some code in the development (post 2.3.0) version that
resets the internal cache of TDB datasets.
That is only a partial fix - the text datasets / Lucene integration is
not going to see that and I don't know if it will work or not.
It would be helpful for someone to try out TDB in Fuseki in Tomcat and
see if the stop/start via the Tomcat manager is better in the current
development build.
You asked
what I'd expect to happen with active transactions. If they are
transactions, I'd expect them to complete or be undone. Really, I just
want the same behavior that occurs with starting and stopping the
complete running Tomcat instance (which works, but it forces me to bring
down other services).
undoing them (aborting) is easy to provide :-)
TDB is safe to just blow the process away - the TDB transactions will
sort it out. (To show I believe that - I stop Fuseki with "kill"
including "kill -9")
By large files (with regard to SPARQL upload vs. bulkloader) I mean
large for an ontology, not large for data. I'm talking about 2 files,
each with about 2.x million triples.
So, I'm now trying to just use sparql update, but I can't get it to
work. I don't get any error, it just hangs indefinitely on a command like:
s-update --service http://localhost:8080/fuseki/sig/update 'CLEAR DEFAULT'
How long did you wait?
The only thing I can think of is that there is another write transaction
somewhere.
I'm thinking that it may be in read-only mode somehow? I've seen docs
mentioning passing the argument --update to the fuseki server. But I'm
not sure how this works if launched via Tomcat. Any suggestions that you
might have here would be much appreciated.
--update is just a way when there isn't a full configuration -- it
causes the default update services to be part of the configuration built
from the command line. A configuration file is a better way to take
detailed control in Tomcat.
Andy
Thanks,
Todd
Landon Todd Detwiler
Structural Informatics Group (SIG)
University of Washington
phone: 206-351-7721
On 9/16/15 12:44 PM, Andy Seaborne wrote:
On 15/09/15 21:00, Todd Detwiler wrote:
Hi Andy,
In this case I am running the version of Jena that came with a recent
(within the last couple of months) Fuseki2 war file distribution. The
core libs look to be 2.13.*.
Fuseki 2.3.0 / Jena 3.0.0 was July.
I can't say what Tomcat does under the hood when you start and stop
apps, but the behavior I saw wasn't just through the text-based http
tools, the same behavior occurs via the manager gui. I know that there
are start-up and shutdown listener hooks in Tomcat. So I could insert
some clean-up code there if I knew what needed to happen.
I don't know about the text index. For TDB, something needs to
release the database (StoreConnection.reset() - caution - what about
active transactions?)
Yes, I could do as you say and limit the update endpoint to localhost.
Will rebuilding the whole store (I want to drop everything in it,
rebuild from two large files, and regenerate the Lucene indexes) be as
efficient as using the bulk loader? Is that your recommended method?
It depends on "large". The TDB bulkloader is faster than just loading
via the usual API. Maybe the same for the text index.
Andy
Your last sentence said,
SPARQL Update (in the latest version where text changes and database
changes are coordinated) is
What did you intend to say after "is"?
Thanks,
Todd
Landon Todd Detwiler
Structural Informatics Group (SIG)
University of Washington
phone: 206-351-7721
On 9/15/15 12:46 PM, Andy Seaborne wrote:
On 15/09/15 17:39, Todd Detwiler wrote:
Let me start with what I'm trying to do. Basically, I'm writing a
script, that will run off of a cron job, that will rebuild my TDB
store
(and Lucene indexes) on a regular basis. I looked into using the TDB
command line scripts, but they fail because another JVM (the tomcat
app)
is already using the TDB store. I looked into using Sparql update
through Fuseki, but I like the idea of the store being read only (and
therefore update is not allowed). So, I am experimenting with
having my
script interact with the Tomcat Manager, stopping the running
Fuseki app
(but not all of tomcat), making changes, and then restarting. I am
able
to stop it fine, using a url like:
(domain):(port)/manager/text/stop?path=/fuseki
However, the corresponding call to start fails:
(domain):(port)/manager/text/start?path=/fuseki
I am getting a stack trace in the tomcat log file that indicates an
issue with file locks related to Lucene indexing. Now, if I stop
tomcat
entirely, and start it again, everything works fine. But I don't
want to
shut down the other services. Do I need to add a listener that
performs
some sort of operation when Tomcat stops? Have others encountered the
same issue. I've included the stack trace at the bottom of this
message.
I'd appreciate any insight anyone could offer. Or let me know if you
think using Sparql update would be a preferred way of rebuilding the
store?
Thanks,
Todd
Hi Todd,
Firstly, which version of jena are you using? There were some changes
to the Lucene indexing around updates that changed the way it works
which may play into this. I see "com.hp.hpl.jena" so it's not the
latest.
Also, I don't know what stopping a webapp in Tomcat actually does. The
standard ServletContextListener only has an operation called
synchronously so it is tricky to clear up in-flight transactions (TDB
does not have forced system aborts by design). Fuseki2, and maybe
wider, needs a proper service lifecycle.
Restarting is clearly calling assembler processing.
Could you protect the update endpoints with the tomcat configuration
to, say, only respond to requests for localhost? While forcing
listening to a specific host isn't very safe, localhost is different
because it means the request was on the loop back interface so can't
be done from outside.
SPARQL Update (in the latest version where text changes and database
changes are coordinated) is
Andy
15-Sep-2015 09:02:55.968 SEVERE [http-nio-8080-exec-10]
org.apache.catalina.core.StandardContext.listenerStart Exception
sending
context initialized event to listener instance of class
org.apache.jena.fuseki.server.FusekiServerListener
com.hp.hpl.jena.assembler.exceptions.AssemblerException: caught:
null
doing:
root: http://localhost/jena_example/#indexLucene with type:
http://jena.apache.org/text#TextIndexLucene assembler class: class
org.apache.jena.query.text.assembler.TextIndexLuceneAssembler
root: http://localhost/jena_example/#text_dataset with type:
http://jena.apache.org/text#TextDataset assembler class: class
org.apache.jena.query.text.assembler.TextDatasetAssembler
at
com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.openBySpecificType(AssemblerGroup.java:138)
at
com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.open(AssemblerGroup.java:117)
at
com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$ExpandingAssemblerGroup.open(AssemblerGroup.java:81)
at
com.hp.hpl.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:52)
at
com.hp.hpl.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:48)
at
org.apache.jena.query.text.assembler.TextDatasetAssembler.open(TextDatasetAssembler.java:63)
at
org.apache.jena.query.text.assembler.TextDatasetAssembler.open(TextDatasetAssembler.java:41)
at
com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.openBySpecificType(AssemblerGroup.java:130)
at
com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.open(AssemblerGroup.java:117)
at
com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$ExpandingAssemblerGroup.open(AssemblerGroup.java:81)
at
com.hp.hpl.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:52)
at
com.hp.hpl.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:48)
at
org.apache.jena.fuseki.build.Builder.buildDataService(Builder.java:76)
at
org.apache.jena.fuseki.build.Builder.buildDataAccessPoint(Builder.java:61)
at
org.apache.jena.fuseki.build.FusekiConfig.servicesAndDatasets(FusekiConfig.java:133)
at
org.apache.jena.fuseki.build.FusekiConfig.readConfigFile(FusekiConfig.java:75)
at
org.apache.jena.fuseki.server.FusekiServer.initServerConfiguration(FusekiServer.java:248)
at
org.apache.jena.fuseki.server.FusekiServer.initializeDataAccessPoints(FusekiServer.java:216)
at
org.apache.jena.fuseki.server.FusekiServerListener.init(FusekiServerListener.java:70)
at
org.apache.jena.fuseki.server.FusekiServerListener.contextInitialized(FusekiServerListener.java:45)
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4729)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5167)
at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at
org.apache.catalina.manager.ManagerServlet.start(ManagerServlet.java:1293)
at
org.apache.catalina.manager.HTMLManagerServlet.start(HTMLManagerServlet.java:673)
at
org.apache.catalina.manager.HTMLManagerServlet.doPost(HTMLManagerServlet.java:221)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.filters.CsrfPreventionFilter.doFilter(CsrfPreventionFilter.java:213)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:614)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1521)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1478)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.OverlappingFileLockException
at
sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1075)
at java.nio.channels.FileChannel.tryLock(FileChannel.java:1155)
at
org.apache.lucene.store.NativeFSLock.obtain(NativeFSLockFactory.java:217)
at org.apache.lucene.store.Lock.obtain(Lock.java:72)
at
org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:702)
at
org.apache.jena.query.text.TextIndexLucene.openIndexWriter(TextIndexLucene.java:121)
at
org.apache.jena.query.text.TextIndexLucene.<init>(TextIndexLucene.java:114)
at
org.apache.jena.query.text.TextDatasetFactory.createLuceneIndex(TextDatasetFactory.java:97)
at
org.apache.jena.query.text.assembler.TextIndexLuceneAssembler.open(TextIndexLuceneAssembler.java:96)
at
org.apache.jena.query.text.assembler.TextIndexLuceneAssembler.open(TextIndexLuceneAssembler.java:47)
at
com.hp.hpl.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.openBySpecificType(AssemblerGroup.java:130)
... 54 more
