Just to clarify this:
> For writers, we use two transactions (both read and write) within the
> same thread,
We create two different datasets:
newEntry.dataset = TDBFactory.createDataset(newEntry.location);
newEntry.dataset.begin(writeAccess ? ReadWrite.WRITE : ReadWrite.READ);
if(writeAccess) {
newEntry.readDataset=TDBFactory.createDataset(newEntry.location);
newEntry.readDataset.begin(ReadWrite.READ);
}
Regarding the long living reader, we have the following SysTxnState,
when the writer starts its work (the three occurrences from today):
[com.hojoki.semantic.component.rdfstore.impl.TDBCacheListener:113]:
There are active transaction (what is NOT supposed to occur). I'll close
each transaction alive. Current state: Active (R=1 W=0) : Finished
(R=48479, WC=7687, WA=0) Queue 4
[com.hojoki.semantic.component.rdfstore.impl.TDBCacheListener:113]:
There are active transaction (what is NOT supposed to occur). I'll close
each transaction alive. Current state: Active (R=1 W=0) : Finished
(R=7964, WC=22, WA=0) Queue 2
[com.hojoki.semantic.component.rdfstore.impl.TDBCacheListener:113]:
There are active transaction (what is NOT supposed to occur). I'll close
each transaction alive. Current state: Active (R=1 W=0) : Finished
(R=1254, WC=200, WA=0) Queue 0
Perhaps this helps
André
On 01.03.2013 11:03, "Dr. André Lanka" wrote:
Hello,
sorry for my late reply. I did some additional tests and tried to avoid
the problem in our systems as far as possible. It turned out that you
are completely right: A long living reader causes the problem. I still
don't know which circumstances provoke the problem. It looks like a race
condition since we do millions of transactions each day and get only a
handful readers that don't finish.
As a work-around I cancel any read transaction before a writer starts
his work. Next, I'll will investigate which thread doesn't come to an
end. Now and then we get query timeouts (5 seconds limit). Would this
"irritate" the transaction management?
Perhaps we misuse the API? : For writers, we use two transactions (both
read and write) within the same thread, as need to access the "original"
(i.e. untouched) model at the end of our write operation. Could this
make trouble?
I'll keep you posted, about the reason of the problem.
Best
André
On 25.02.2013 23:16, Andy Seaborne wrote:
On 25/02/13 09:05, "Dr. André Lanka" wrote:
Hello,
we encounter some problems while writing to TDB stores (0.9.4). We get
some StackOverflowExceptions now and then. Many objects of
BlockMgrJournal seems to overlay each other.
The problem seems to be solved by calling
StoreConnection.getExisting(Location).flush()
periodically.
We wonder, if there is also an internal heuristic that determines when
the journal files get transferred to the base files or if manually
flushing is always necessary?
Do you have a sample stack trace? I can't tell what going on from the
description.
You get a stack of BlockMgrJournals under two conditions:
1/ Changes from multiple writers are batched together and the
transactions only applied to the permanent database every few writers
commits. This is not official API but you can, currently set this
TransactionManager.QueueBatchSize
the default is 10.
(Fuseki sets it to zero for increased stability when running many
datasets in one server)
I would not have thought this on it's own would cause stack overflow
2/ But if there is a read transaction about, the writer changes can't be
written back (it's a write-ahead logging system).
If you have long-lived reader, then they may be blocking the writer
commits finally clearing up (this is after the commit of the
transaction).
> StoreConnection.getExisting(Location).flush()
Hmm - looking at that, it needs a "synchronized" (on the Transaction
manager underlying operation)
Andy
Thanks
André
--
Dr. André Lanka * 0178 / 134 44 47 * http://dr-lanka.de
