[
https://issues.apache.org/jira/browse/JENA-327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464023#comment-13464023
]
Simon Helsen commented on JENA-327:
-----------------------------------
ok, so if we write out n-quads in a read transaction and that takes a long
time, TDB/Tx will keep serving queries and permitting updates, but it won't
merge the journal during that time (to honor the transactional semantics). What
about memory usage? I am trying to assess whether this approach would not cause
other dangers to a running system. And of course, a restore in such cases will
take as long as it takes to import the quads. I did some quick calculations and
a store with 50 million quads would take 15 minutes on my machine, probably
faster on decent server hardware.
> TDB Tx transaction lock to permit backups
> -----------------------------------------
>
> Key: JENA-327
> URL: https://issues.apache.org/jira/browse/JENA-327
> Project: Apache Jena
> Issue Type: Improvement
> Components: TDB
> Affects Versions: TDB 0.9.4
> Reporter: Simon Helsen
>
> With large repositories, it is important to be able to create backups once in
> a while. This is because recreating an rdf store with millions of triples can
> be forbiddingly expensive. Moreover, it should be possible to take those
> backups while still allowing read activity on the store as in many cases, a
> complete shutdown is usually not possible. Before the introduction of tx, it
> was relatively straightforward to provide the right locks on the client-side
> to safely suspend any disk activity for a period of time enough to make a
> backup of the index.
> However, since tx, things have become slightly more complicated because TDB
> Tx touches the disk at other times than when performing write/sync
> activities. Right now, because of some understanding of how TDB Tx is
> implemented, it is still possible for clients to avoid disk activities to
> implement a backup process, but this dependency on TDB Tx implementation
> details is not very good. Moreover, we anticipate that in the future, the
> merging process from the journal into the main index may become entirely
> asynchornous for performance reasons. The moment that happens, client have no
> control anymore as to when the disk is being touched.
> For this reason, we are requesting the following feature: a "backup" lock (by
> lack of a better name). Its semantics is that when the lock is taken, TDB Tx
> guarantees that no disk activity takes place and if necessary pauses
> activities. In other words, no write transaction should be able to complete
> and read transactions will not attempt to merge the journal. The idea would
> be that regular read activities can still continue. The API could be as
> simple as something like this:
> try {
> dataset.begin(ReadWrite.BACKUP) ;
> <do whatever is necessary to backup the index>
> } finally {
> dataset.end()
> }
> As for the implementation, we suspect you currently have locks in place which
> could be used to guarantee this behavior. E.g. could
> txn.getBaseDataset().getLock().enterCriticalSection(Lock.WRITE) be sufficient?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
