On 11/08/16 14:10, A. Soroka wrote:
I'm probably missing something obvious here (and this won't help
Mikael Pesonen immediately even if it does make sense), but would it be
useful to have a two-journal system with a swap operation? So while a
thread is loading the first journal, another is emptying it into the
main store, and then you swap to get an empty journal, etc. The idea
would be that instead of the overflow, you'd get blocking when the
filling journal is full and the emptying journal is still dumping. I'm
sure I'm missing something here. {grin}
That would be interesting to make the work of putting the journal in the
main database not be part of some transaction or other. In TDB, it can
even be part of a reader (its the first time any transaction can flush
the queue).
The problem that Mikael seems to have encountered is that there is never
a moment when the main database is free to update - readers are looking
all the time.
Because TDB is based on a journal that is a "redo" log, changes are not
made the main database during the transaction. They happen in a
in-memory overlay and also recorded in the journal. One layer per
writer not flushed from the journal.
What's blocking the journal being flushed could be another writer but
from the description (it's Fuseki backed so there is not batching of
commits in to large units - see Dick Murray's setup), writing is
sequential and its concurrent readers that don't stop the writers but do
stop the journal clear up.
It's a few 100 per transaction so it does no run to of heap - each layer
however, adds to stack on access.
Andy
