I am not as familiar as Andy is with the transaction subsystem of TDB, but the following is my understanding. If you are interested, the relevant code is in the TransactionManager and Transaction classes.
On Tue, Apr 30, 2013 at 11:39 AM, David Jordan <david.jor...@sas.com> wrote: > > Questions relative to > http://jena.apache.org/documentation/tdb/tdb_transactions.html > The page states that read transactions do not have any locking or other > overhead. In the case of write transactions, it reads as if for every update, > a before and after image of the data is written to a separate journal log. It > does not say this explicitly, is this true? TDB uses Multi-Version Concurrency Control (MVCC). This is sometimes called snapshot isolation (which is TDB's case is equivalent to serializable because only a single writer is allowed). Readers do not block writers, and writers do not block readers. Writers will still block each other. Further, TDB uses block-level MVCC (as apposed to triple/quad level). TDB uses a redo only write-ahead logging system. When a writer changes a block, a copy is made in-memory that only the writer has access to. A pointer to these in-memory blocks is maintained by the write transaction. That in-memory block is then updated with any new values. The writer completes in one of two ways: 1) Abort - The in-memory blocks are simply dropped 2) Commit - The writer logs all modified in-memory blocks to the journal. A commit record is written to the journal. Journal is flushed to disk. After a commit occurs on a write transaction, one of two possibilities exist. If there are no current active transactions, then the modified in-memory blocks are copied back over the existing blocks. If there are active transactions, then the write transaction is placed in a queue for later merging back into the main database. Thereafter, whenever a transaction finishes it checks to see if it is the only active transaction. If it is the only active transaction, then the queue is written back to the base dataset. Now, there is also an additional wrinkle in that there is a "view" of the underlying dataset that is kept and shared between readers on the same "level" (i.e. there is no intervening write transaction that has been flushed back to the dataset). The flushing mechanism above will reset this. I don't quite understand what is going on with this object. A side-effect of this method of performing transactions is the fact that if you have a very long running read transaction, it can cause the commit queue to grow unbounded. At some point you will run out of heap memory, and then the process will crash with an OutOfMemoryException (somewhere, randomly, in the code, not necessarily in one of your writer transactions). > A read transaction would only see the state of the database as it existed > when the transaction was started. If one or more write transactions are > committed during a read transaction, the doc does not state when these > updates are propagated to the database. If they are propagated before the > read transaction finishes, then a before image of the data must be kept and > provided to the read transaction. These before images must be timestamped, > because multiple update transactions may have committed, serially. As one or > more update transactions commit, but before they are propagated to the > database, the transactionally consistent state of the database as perceived > by other transactions is entirely dependent on when they started. In fact, > the state can vary among the different transactions based on both their own > start time, and the time when updates were committed to the database (as > opposed to when the updates are propagated to the database). For example, > assume update transactions UT1 and UT2 are run serially. First UT1 updates > data X and performs a commit. Then UT2 begins and reads X, it should see the > update made by UT1, even if UT1's updates have not yet been propagated to the > database. The doc says that "If a single read transaction runs for a long > time when there are many updates, the system will consume a lot of temporary > resources." I believe this is due to the before/after image journal. Is the > above description correct? As mentioned above, transactions get a "snapshot" of the database at the time they begin. So any other updates occurring in a simultaneous transaction will never be visible to it. If a transaction successfully commits, and you receive notification of that (i.e. the .commit() method returns), then any other transactions started after that will see modifications by the first transaction. When I say after, I specifically mean the first transaction "happens-before", which will occur naturally in a single-threaded program, but must be enforced with some type of synchronization if your program is multi-threaded. The statement about "a lot of temporary resources" is explained above: basically the commit queue can grow unbounded. > In prototyping, Jena "works" without any transactions. If you have a database > where you are running code serially, you can read the database, write to the > database, without using any transactions at all. Obviously, if you have a > series of updates but you encounter an exception that terminates your code > before it completes, you run the risk of corrupting the database. You also > lose the ability of having concurrent transactions, independently doing reads > and/or writes with full isolation. Trying to do this would likely lead to a > corrupted database. But once you want concurrency, updates, and isolation, > transactions become a necessity. Another huge (I would say the most important) reason for running with transactions is because it provides durability guarantees (the D in ACID). If you program crashes while performing an update, the database will be able to recover all statements up to the last successful commit by replaying the journal. > In response to an issue I had yesterday, Andy indicated that the required > order of calls to Jena are the following: > Dataset dataset = TDBFactory.assembleDataset(TDB_ASSEMBLER_FILE); > try { > dataset.begin(ReadWrite.WRITE); // or READ > Model model = dataset.getNamedModel("modelname"); > OntModel omodel = ModelFactory.createOntologyModel( > OntModelSpec.OWL_DL_MEM_RULE_INF, model); > omodel.prepare(); > // perform operations with the model/omodel... > // call dataset.commit() or dataset.abort() > } finally { > dataset.end(); > } > Presumably, one can perform a series of transactions, as follows: > dataset.begin > dataset.commit > dataset.begin > dataset.commit > But if the creation of the Model and OntModel need to be done inside of the > begin/commit, then the extensive overhead of performing the inferencing must > be performed within each transaction. Obviously, if there are updates going > on concurrently, this would be the only way for the second transaction above > to pick up any updates performed concurrently. > But if one "knows" in their system implementation that a given Model (named > graph in the dataset) is read-only, that updates will not be performed, is it > OK to not use transactions at all? The doc does state you can use a > TDB-backed database non-transactionally. I do not know the answer of the best way to use inferencing with transactions. I think that you may indeed have to pay the price of performing the inferencing on each transaction, because those inferences are only held in memory, and are not built into the transaction system. > If READ transactions truly incur no overhead, is there any reason NOT to > enclose them in transactions anyway? And if a given Model is never updated, > can one perform a series of READ transactions without incurring the overhead > of the OntModel inferencing with each READ transaction? As Andy had pointed > out in response to me, the Model and OntModel need to be done in the context > of a transaction, which implies they cannot span across transactions. But for > READ transactions, or no transactions at all, I have been able to > successfully perform them without a problem. > The doc states that "A TDB-backed dataset can be used non-transactionally but > once used in a transaction, it should be used transactionally after that." Is > this due to how TDB determines the latest state of the values? Does this rule > only apply within a single JVM use of a TDB dataset, or does it apply even > across separate JVM interactions with the TDB dataset? You must not mix transaction and non-transactional access to the dataset. Non-transactional access relies on the client code enforcing Multiple Reader / Single Writer locking (MRSW). If you want to use transactions, you must also have the reads inside of a transaction. > What I am struggling with here is the need and benefit of being able to > provide read access to named graphs that are created once and not changed, > but these named graphs live in the same TDB database as other named graphs > that will have updated transactions and require transactions. Is there an > answer to this? > Hope this helps. My recommendation is that you always use transactions unless your database is ephemeral and you plan to create a new one every time your application starts up. If you want to persist any data between runs of your application (even after crashes), then you need the durability guarentee that transactions provide. -Stephen P.S. Andy, please chime in where I've gotten something wrong (as I undoubtedly have, it's a pretty complex bit of the code)
