Hi, our app is based on Drools Guvnor and our repository is currently about
20GB. Every once in a while, we will build up enough indexes that it has a to
do a very big merge (around 4GB). However, at a certain point the
TransientRepository shuts down for whatever reason (I'm new to Jackrabbit and
haven't been able to track down why...I am still logged into the app when this
occurs) and tries to stop the IndexMerger. When the IndexMerger finally does
finish, it says merging has been canceled and that it is deleting the index,
but looking at the index directory I can see that the folder is still present.
Then the TransientRepository starts up again it retries and starts the whole
cycle over again, resulting in a bunch of leftover 4GB index folders and
eventually a full hard drive.
We initially encountered this problem on Jackrabbit 1.4.5 (came with Guvnor
when this project was started) and have recently upgraded to 2.8.0 thinking it
would help, but we were still able to reproduce the issue. Is this known and/or
expected behavior with the TransientRepository? Is there anything in Jackrabbit
itself that causes all Sessions to spontaneously close and shut down the
repository? If so how do we work around this?
Below is an excerpt from our most recent run on 2.8.0. I forgot to enable debug
statements on TransientRepository, but if needed I can provide the 1.4.5 logs
that have them.
2014-07-16 16:00:38,890 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] requesting merge for
[_10w6:99896, _10ze:99781, _115x:99915, _11qc:99849, _12kx:99861, _13mh:43521,
_14ha:34129, _15c5:34256, _1672:34254, _16ur:29993]
2014-07-16 16:00:38,891 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] # of busy merge workers: 1
2014-07-16 16:00:54,095 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] dispose IndexMerger
2014-07-16 16:00:54,097 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] quit flag set
2014-07-16 16:06:25,673 INFO
[org.apache.jackrabbit.core.query.lucene.IndexMerger] merged 675455 documents
in 344310 ms into _16uy.
2014-07-16 16:06:25,674 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] index merging canceled
2014-07-16 16:06:25,675 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] deleting index _16uy
2014-07-16 16:06:25,675 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] IndexMerger.Worker thread
stopped
2014-07-16 16:06:25,676 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] Worker finished
2014-07-16 16:06:26,539 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] dispose IndexMerger
2014-07-16 16:06:26,540 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] quit flag set
2014-07-16 16:06:34,731 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] requesting merge for
[_10w6:99896, _10ze:99781, _115x:99915, _11qc:99849, _12kx:99861, _13mh:43521,
_14ha:34129, _15c5:34256, _1672:34254, _16ur:29993]
2014-07-16 16:06:34,732 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] # of busy merge workers: 1
2014-07-16 16:06:54,192 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] dispose IndexMerger
2014-07-16 16:06:54,193 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] quit flag set
2014-07-16 16:12:28,267 INFO
[org.apache.jackrabbit.core.query.lucene.IndexMerger] merged 675455 documents
in 352127 ms into _16uz.
2014-07-16 16:12:28,270 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] index merging canceled
2014-07-16 16:12:28,270 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] deleting index _16uz
2014-07-16 16:12:28,271 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] Worker finished
2014-07-16 16:12:28,271 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] IndexMerger.Worker thread
stopped
2014-07-16 16:12:28,607 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] dispose IndexMerger
2014-07-16 16:12:28,608 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] quit flag set
2014-07-16 16:12:34,629 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] requesting merge for
[_10w6:99896, _10ze:99781, _115x:99915, _11qc:99849, _12kx:99861, _13mh:43521,
_14ha:34129, _15c5:34256, _1672:34254, _16ur:29993]
2014-07-16 16:12:34,631 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] # of busy merge workers: 1
2014-07-16 16:12:54,220 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] dispose IndexMerger
2014-07-16 16:12:54,221 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] quit flag set
2014-07-16 16:18:23,807 INFO
[org.apache.jackrabbit.core.query.lucene.IndexMerger] merged 675455 documents
in 348109 ms into _16v0.
2014-07-16 16:18:23,810 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] index merging canceled
2014-07-16 16:18:23,810 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] deleting index _16v0
2014-07-16 16:18:23,810 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] IndexMerger.Worker thread
stopped
2014-07-16 16:18:23,812 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] Worker finished
2014-07-16 16:18:24,165 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] dispose IndexMerger
2014-07-16 16:18:24,166 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] quit flag set
2014-07-16 16:18:31,090 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] requesting merge for
[_10w6:99896, _10ze:99781, _115x:99915, _11qc:99849, _12kx:99861, _13mh:43521,
_14ha:34129, _15c5:34256, _1672:34254, _16ur:29993]
2014-07-16 16:18:31,091 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] # of busy merge workers: 1
2014-07-16 16:18:54,236 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] dispose IndexMerger
2014-07-16 16:18:54,238 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] quit flag set
2014-07-16 16:22:07,872 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] deleting index _16v1
2014-07-16 16:22:08,545 ERROR
[org.apache.jackrabbit.core.query.lucene.IndexMerger] Error while merging
indexes:
java.io.IOException: There is not enough space on the disk
at java.io.RandomAccessFile.setLength(Native Method)
at
org.apache.lucene.store.FSDirectory$FSIndexOutput.setLength(FSDirectory.java:489)
at
org.apache.lucene.index.CompoundFileWriter.close(CompoundFileWriter.java:195)
at
org.apache.lucene.index.SegmentMerger.createCompoundFile(SegmentMerger.java:137)
at
org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:3219)
at
org.apache.jackrabbit.core.query.lucene.PersistentIndex.addIndexes(PersistentIndex.java:114)
at
org.apache.jackrabbit.core.query.lucene.IndexMerger$Worker.run(IndexMerger.java:526)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
2014-07-16 16:22:08,551 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] Worker finished
2014-07-16 16:22:08,552 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] IndexMerger.Worker thread
stopped
2014-07-16 16:22:08,890 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] dispose IndexMerger
2014-07-16 16:22:08,891 DEBUG
[org.apache.jackrabbit.core.query.lucene.IndexMerger] quit flag set
Any help would be greatly appreciated.
Thanks,
David Hsu
