Andy, I was probably not all that clear. I admit that I was initially suspecting increased lock contention, but I don't think those are the cause here because our internal regular (functionally-oriented and mostly single-threaded) junit suite slowed down by 25%. When I looked at the underlying stats (which we always collect in our framework), I noticed that the total suite index write time took 3 times as long. I did this by comparing Tx and nonTx because I know that only last week with the May 15 TDB build, this junit suite was running at about the same speed in both tx and non-tx. (I am not claiming tx is as fast as non-tx in all circumstances, but we had observed significantly faster index write times under almost all circumstances)
Let me stress that these are not even performance tests. I only executed this experiment because our internal client adopted a new build of ours with the latest TDB build (last weekend, the release candidate) and suddenly reported the 40% slowdown on both index and query compared to the previous TDB (tx) build of May 15 for 25 concurrent users in a write-heavy test scenario. So, the question is how we can identify the culprit. Perhaps there is even a legitimate reason, but Laurent seems to have observed the same with respect to the 0.9.0 build. Could we not use some of the existing Jena unit tests which verify concurrent query/write behavior in Tx? Perhaps increasing the load to get more useful execution times and then run them on the SVN state of May 15 and HEAD and compare. I am not sure if it would narrow down the problem, but at least it would demonstrate the regression Simon PS: Is this concern not a good reason to delay the release, at least until it is understood? From: Andy Seaborne <[email protected]> To: [email protected] Date: 06/13/2012 06:08 PM Subject: Re: concurrency and slower results On 13/06/12 18:16, Simon Helsen wrote: > I have to correct my statement below, I was misreading my own numbers. > The overall write time was about 3 times slower, not just 70%. This is > comparing the latest build, once with and once without transactions > (single user). The numbers (40%) I mentioned before came from 25 > concurrent users on a build using transactions from May 15 compared to a > build using transactions from last weekend. That is not quite the same > kind of comparison of course. However, I do recall (I don't have that on > paper) that my own test suite with the May 15 build was running at a > comparable speed whether I was using transactions or not. So I dare to > conclude that write transactions are now considerable more expensive > than around May 15 Is there some way we can turn this into a publicly usable example? The recent changes to fix problems (JENA-250, JENA-252) might change the concurrency but the changes also added a potential file system sync the dataset when transitioning at first use transactionally. Depending on how you use StoreConenction.release, that might have an effect but in normal use it's not supposed to (and also syncs should only go to the OS if they are needed as components keep a flag as to whether a raw update has been done. I'm a little surprised that tx and non-tx were the same speed in the first place. > by lock contention on the writes For clarity: TDB does not use 2-phase locking - it uses a write-behind log. It does use java locks to protect key datastructures. Andy
