I'm starting to look at CDV-392
(https://jira.terracotta.org/jira/browse/CDV-392) which is meant to abstract
common eviction logic from the sessions product, the ehcache config module, and
the Terracotta Cache/JBoss TreeCache labs project. The rest of this email is a
summary of a meeting on this today.
The ehcache stuff started from the session eviction code and still uses very
similar data structures so these two should be able to share an abstraction
fairly easily. The jboss stuff (let jboss == Terracotta Cache for brevity)
shares no common heritage with the others but is not very sophisticated.
== Sesssions ==
The session eviction code is mostly in
TerracottaSessionManager.SessionInvalidator (the thread doing the eviction) and
SessionDataStore (that holds the session state). Each node in the cluster has
a SessionInvalidator thread and each will periodically awake to try to evict
expired sessions. To avoid collision problems, all of these threads lock on a
shared DSO lock across the cluster, so that only one of the invalidation
threads runs at a time.
In SessionDataStore, there is a main store and a timestamp store. Both are
keyed on session IDs; the main store has SessionData as a value and the
timestamp store has Timestamps as the values. The SessionInvalidator uses the
timestamp store to determine whether it's even worth looking at the main store
for a session if it's not possible for it to be expired yet. That minimizes
the need for faulting stuff across nodes that isn't even eligible in the first
place.
However, this faulting across nodes will still occur when real expiration
occurs as the invalidator thread is only running on one node and looks at all
sessions.
A sessions-specific thing is that if someone is using the session, we cannot
invalidate, even if the session is past its idle time, so those sessions are
not considered. This is checked with a dso trylock.
Things that could be improved here are increasing the concurrency (not locking
across all the nodes) and reducing the faulting that occurs on expiration.
== ehcache ==
ehcache eviction stuff started from the sessions code and diverged to make it
perform better for eviction. It has similar data structures of the main data
map and timestamp map. The difference is that there are both local evictions
(keys created on this node) and global evictions (keys created on a node that's
gone). Local evictions still lock a shared lock but should do less faulting by
just looking at local stuff. When local is done, if time remains, a global
eviction will occur.
ehcache is a little different in that instead of 1 lock per session, data is
striped across a set of maps similar to ConcurrentHashMap. The number of maps
is a config parameter ("concurrent"). The eviction threads are actually a pool
(configurable size) and each thread will be assigned to concurrency/pool_size
number of data maps.
These seem like good changes that can be pulled back and used in the session
eviction as well to reduce faulting.
== jboss ==
This cache is a little different as it is not a flat key/value space. The
cache is a tree defined by FQN (fully-qualified names). Each FQN can have a
different eviction policy and this is highly configurable.
== design ==
Some things to shoot for in CDV-392:
- increased concurrency - want to improve the ability of multiple nodes to do
eviction without a global lock if possible. Local invalidation is a good
technique here.
- reduced faulting - want to minimize the amount of faulting that must occur to
do eviction, the ehcache changes seem like a good start
- everything must ultimately be evictable - need to make sure that anything can
ultimately be evicted, even if the client has gone down, etc. ehcache changes
seem to cover this as well.
== next steps ==
I'm going to send out notes (this email) and put together a proposal, which may
be in the form of notes or code or whatever makes sense and get some feedback
next week.
_______________________________________________
tc-dev mailing list
[email protected]
http://lists.terracotta.org/mailman/listinfo/tc-dev
