In abtract
I am load testing a Hibernate/JBC3 web application and am seeing that about 7%
of the time inserting a child entity into a previously-empty collection results
in update (INSERT) into DB but local cache remains unchanged:
-collection Node is still empty
-there is no child Entity node
This does not appear to be a race condition, as from that point on in any TX
parent.getChildren() will return an empty collection from cache (even though DB
state is non-empty).
This seems vaguely similar to https://jira.jboss.org/jira/browse/JBCACHE-1481
but that issue's marked resolved.
In detail
App Config
Single Tomcat 6 NIO
Hibernate 3.3.1.GA
JBossCache 3.0.3.GA - Local Cache only; using default Locking (MVCC);
READ_COMMITTED isolation
BTM JTA transaction manager
Load Test
JMeter load test where each of userCount users loop over the following scenario
of HTTP requests:
for(I iterations)
| {
| login();
| thinkTime //sleep;
| getRestrictedData(); //requires being logged in
| thinkTime //sleep;
| logout();
| thinkTime //sleep;
| }
|
App Pseudo Code
Domain Model
-Transactional entity caches User, UserSession; collection cache User.sessions
| class User{
| private long id;
| private Set<UserSession> sessions; //1-to-[0,1] modeled as one-to-many
with unique constraint
|
| public boolean isLoggedIn()
| {
| return !getSessions().isEmpty();
| }
|
| public void addSession(UserSession us)
| {
| us.setUser(this);
| sessions.add(us);
| }
|
| public Set<UserSessions> getSessions()
| {
| return sessions;
| }
| }
|
|
|
| class UserSession
| {
| private User user;
|
| void setUser(User usr)
| {
| this.user = usr;
| }
|
| }
Service Methods
//insert new UserSession if credentials check out
| //throw exception otherwise
| public void login(id, credentials)
| {
| User usr = dao.getUser(id);
| if(credentials == bad)
| {
| throw new AuthenticationException();
| }
| usr.add(new UserSession());
| }
|
| //return data only if a userSession is present
| //throw exception otherwise
| public String getRestrictedData(id)
| {
| User usr = dao.getUser(id);
| if(! usr.isLoggedIn())
| {
| throw new AuthorizationException("not logged in");
| }
| return data;
| }
|
| public void logout()
| {
| //clear userSessions
| }
| }
|
Problem Observed
When userCount==1, no requests fail, regardless of thinkTime (varied latter
between 0 and 2000ms)
| When userCount is > 30, about 10-20% of users eventually (and most of the
time on the very 1st iteration), fail like so:
| -login() //success
| -getRestrictedData() //fail: user not logged in
|
| My load test terminates on 1st failure. Inspecting states of DB and JBoss
Cache (via JMX) reveal:
| -DB does have record of UserSession in question
| -In Collection cache: User.sessions Node contains an empty PersistentSet
| -In entity cache: User node references above (empty) collection; no node
for UserSession that is in DB;
| -hibernateVersion of User (parent) is updated in DB but not in Cache node
|
| Musings
|
| Though I am unable to repro this problem with 1 user, it is not possible
for user A to affect state of user B in my app. So I am not quite clear on
what's going on here. Since I don't (yet) understand the internals of MVCC (and
jbc in general), I'd appreciate any thoughts and answers to the following
questions that'll help me narrow down the problem:
|
| 1. In MVCC, how many threads participate in login() method? Is it the case
that one thread does the read and writes a versioned update, but another thread
merges the updated data *after* TX commit?
|
| 2. A somewhat dated http://www.jboss.org/community/docs/DOC-10266 wiki says
there exists a faulty assumption about consistent ordering of synchronizations
in Hibernate/JBC (problem #5). Has that been addressed? If not, could that be
causing given problem?
|
| 3. Same wiki's problem #7:
| anonymous wrote : Hibernate should either retry or JBossCache should fail
silently on this. (Nikita: not sure what 'this' refers to) As long as the data
is stored in the db correctly, failure putting the data in the cache could fail
silently. Next time someone requests the entity, it'd be retrieved from db and
put in the cache.
|
| This paragraph seems to imply that it is normal that cache is not updated
on insert of new UserSession during login(). However, it also expects
subsequent User.getSessions() to result in a DB read (and a put into cache).
But what I am seeing that User.getSessions() in getRestrictedData() results in
a cache hit on a (stale) node containing empty set user.sessions. Should
addSession() in login() result in User.sessions collection node being
invalidated (to be populated during subsequent reads) or being updated with a
new reference to UserSession node? Similarly, should addSession() result in put
into UserSession entity cache, or is that put expected to occur only on
subsequent read?
|
| 4. Finally, turning up debugging shows multitude of these 2 types of
messages. These are emitted seemingly in failing and succeeding requests:
|
| anonymous wrote : [3/2/09 16:27:38:268] DEBUG
[d585d9d4-cc59-4b35-b717-d26c97db8f89,ltester-100000]
org.jboss.cache.interceptors.InvocationContextInterceptor - FAIL_SILENTLY
Option is present - suspending any ongoing transaction.
| | [3/2/09 16:27:38:269] DEBUG
[76225fe7-1541-4960-9c2d-cdbe1330f5c5,ltester-100001]
org.jboss.cache.invocation.CacheInvocationDelegate - putForExternalRead()
called with Fqn
/com/doppelganger/domain/User/userSessions/COLL/com.doppelganger.domain.User.userSessions#183
and this node already exists. This method is hence a no op.
| |
|
| Does the latter say that a collections cache node will not be updated
because one already exists. What if existing node contains an empty collection
but the candidate node contains a non-empty collection?
View the original post :
http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4214380#4214380
Reply to the post :
http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=4214380
_______________________________________________
jboss-user mailing list
[email protected]
https://lists.jboss.org/mailman/listinfo/jboss-user
