[ 
https://issues.apache.org/jira/browse/GEODE-7763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039240#comment-17039240
 ] 

Barrett Oglesby commented on GEODE-7763:
----------------------------------------

Heap histogram comparisons on 2.2.2.RELEASE and master show different numbers 
of live DeltaCapableGemFireSession instances during the test.

2.2.2.RELEASE:
{noformat}
 num     #instances         #bytes  class name
----------------------------------------------
 133:            72           3456  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
 157:            45           2160  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
 181:            33           1584  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
 181:            32           1536  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
 141:            57           2736  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
 145:            55           2640  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
 157:            44           2112  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
 154:            47           2256  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
 179:            34           1632  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
 154:            48           2304  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
{noformat}
master:
{noformat}
 num     #instances         #bytes  class name
----------------------------------------------
  65:           254          12192  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
  65:           250          12000  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
  64:           249          11952  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
  64:           274          13152  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
  65:           265          12720  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
  65:           243          11664  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
  65:           261          12528  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
  65:           255          12240  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
  65:           258          12384  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
  65:           255          12240  
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession
{noformat}
Thread dumps on the client show why that is.

2.2.2.RELEASE

A thread dump shows 180 Session Threads.

86 Session Threads are BLOCKED during deepCopy of a future get response in 
GemFireSession.hasDelta here:
{noformat}
"Session Thread 167" #202 daemon prio=5 os_prio=31 tid=0x00007f857ebb5000 
nid=0x1c203 waiting for monitor entry [0x0000700013270000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSession.hasDelta(AbstractGemFireOperationsSessionRepository.java)
        - waiting to lock <0x00000007a9b88118> (a 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$SpyingDataSerializableSessionSerializer.serialize(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:467)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$SpyingDataSerializableSessionSerializer.serialize(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:441)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.lambda$toData$0(AbstractDataSerializableSessionSerializer.java:91)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer$$Lambda$435/306742862.apply(Unknown
 Source)
        at java.util.Optional.map(Optional.java:215)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.toData(AbstractDataSerializableSessionSerializer.java:90)
        at 
org.apache.geode.internal.InternalDataSerializer.writeUserObject(InternalDataSerializer.java:1625)
        at 
org.apache.geode.internal.InternalDataSerializer.writeWellKnownObject(InternalDataSerializer.java:1587)
        at 
org.apache.geode.internal.InternalDataSerializer.basicWriteObject(InternalDataSerializer.java:2238)
        at org.apache.geode.DataSerializer.writeObject(DataSerializer.java:2953)
        at org.apache.geode.CopyHelper.doDeepCopy(CopyHelper.java:241)
        at org.apache.geode.CopyHelper.copy(CopyHelper.java:190)
        at 
org.apache.geode.internal.cache.LocalRegion.conditionalCopy(LocalRegion.java:1606)
        at 
org.apache.geode.internal.cache.LocalRegion.nonTxnFindObject(LocalRegion.java:1519)
        at 
org.apache.geode.internal.cache.LocalRegionDataView.findObject(LocalRegionDataView.java:188)
        at 
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1443)
        at 
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1377)
        at 
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1362)
        at 
org.apache.geode.internal.cache.AbstractRegion.get(AbstractRegion.java:416)
        at 
org.springframework.data.gemfire.GemfireTemplate.get(GemfireTemplate.java:166)
        at 
org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.findById(GemFireOperationsSessionRepository.java:95)
        at 
org.springframework.session.data.gemfire.AbstractGemFireIntegrationTests.get(AbstractGemFireIntegrationTests.java:405)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.lambda$newAddSessionAttributeTask$2(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:220)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$$Lambda$474/775341308.call(Unknown
 Source)
{noformat}
24 Session Threads are waiting for a future here:
{noformat}
"Session Thread 5" #40 daemon prio=5 os_prio=31 tid=0x00007f857a9d9800 
nid=0x7203 waiting on condition [0x0000700008e8a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000079a4e03d8> (a 
java.util.concurrent.CountDownLatch$Sync)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at 
org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:61)
        at 
org.apache.geode.internal.util.concurrent.FutureResult.get(FutureResult.java:69)
        at 
org.apache.geode.internal.cache.LocalRegion.nonTxnFindObject(LocalRegion.java:1503)
        at 
org.apache.geode.internal.cache.LocalRegionDataView.findObject(LocalRegionDataView.java:188)
        at 
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1443)
        at 
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1377)
        at 
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1362)
        at 
org.apache.geode.internal.cache.AbstractRegion.get(AbstractRegion.java:416)
        at 
org.springframework.data.gemfire.GemfireTemplate.get(GemfireTemplate.java:166)
        at 
org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.findById(GemFireOperationsSessionRepository.java:95)
        at 
org.springframework.session.data.gemfire.AbstractGemFireIntegrationTests.get(AbstractGemFireIntegrationTests.java:405)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.lambda$newAddSessionAttributeTask$2(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:220)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$$Lambda$474/775341308.call(Unknown
 Source)
{noformat}
18 Session Threads are in CopyHelper.doDeepCopy during a get here:
{noformat}
"Session Thread 1" #36 daemon prio=5 os_prio=31 tid=0x00007f857a9af800 
nid=0x6d03 runnable [0x0000700008a7d000]
   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.put(HashMap.java:611)
        at java.util.HashSet.add(HashSet.java:219)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSessionAttributes.lambda$sessionAttributesChangeInterceptor$0(AbstractGemFireOperationsSessionRepository.java:1151)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSessionAttributes$$Lambda$488/179123654.apply(Unknown
 Source)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes.doSetAttribute(AbstractGemFireOperationsSessionRepository.java:1325)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes.setAttribute(AbstractGemFireOperationsSessionRepository.java:1313)
        - locked <0x00000007a647d450> (a 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes.lambda$from$2(AbstractGemFireOperationsSessionRepository.java:1416)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes$$Lambda$455/1987823026.accept(Unknown
 Source)
        at java.util.HashMap$KeySet.forEach(HashMap.java:932)
        at 
java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes.from(AbstractGemFireOperationsSessionRepository.java:1415)
        - locked <0x00000007a647d450> (a 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionSerializer.deserialize(DataSerializableSessionSerializer.java:156)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$SpyingDataSerializableSessionSerializer.deserialize(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:476)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$SpyingDataSerializableSessionSerializer.deserialize(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:441)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.fromData(AbstractDataSerializableSessionSerializer.java:107)
        at 
org.apache.geode.internal.InternalDataSerializer.readUserObject(InternalDataSerializer.java:1786)
        at 
org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2878)
        at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2978)
        at org.apache.geode.CopyHelper.doDeepCopy(CopyHelper.java:242)
        at org.apache.geode.CopyHelper.copy(CopyHelper.java:190)
        at 
org.apache.geode.internal.cache.LocalRegion.conditionalCopy(LocalRegion.java:1606)
        at 
org.apache.geode.internal.cache.LocalRegion.nonTxnFindObject(LocalRegion.java:1519)
        at 
org.apache.geode.internal.cache.LocalRegionDataView.findObject(LocalRegionDataView.java:188)
        at 
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1443)
        at 
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1377)
        at 
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1362)
        at 
org.apache.geode.internal.cache.AbstractRegion.get(AbstractRegion.java:416)
        at 
org.springframework.data.gemfire.GemfireTemplate.get(GemfireTemplate.java:166)
        at 
org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.findById(GemFireOperationsSessionRepository.java:95)
        at 
org.springframework.session.data.gemfire.AbstractGemFireIntegrationTests.get(AbstractGemFireIntegrationTests.java:405)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.lambda$newAddSessionAttributeTask$2(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:220)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$$Lambda$474/775341308.call(Unknown
 Source)
{noformat}
52 Session Threads are in the 
SessionEventHandlerCacheListenerAdapter.afterCreate processing the put 
operation here:
{noformat}
"Session Thread 2" #37 daemon prio=5 os_prio=31 tid=0x00007f8579c81800 
nid=0x6f03 runnable [0x0000700008b80000]
   java.lang.Thread.State: RUNNABLE
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes.getMap(AbstractGemFireOperationsSessionRepository.java)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes.doSetAttribute(AbstractGemFireOperationsSessionRepository.java:1320)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes.setAttribute(AbstractGemFireOperationsSessionRepository.java:1313)
        - locked <0x000000079ef70bd0> (a 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes.lambda$from$2(AbstractGemFireOperationsSessionRepository.java:1416)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes$$Lambda$455/1987823026.accept(Unknown
 Source)
        at java.util.HashMap$KeySet.forEach(HashMap.java:932)
        at 
java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes.from(AbstractGemFireOperationsSessionRepository.java:1415)
        - locked <0x000000079ef70bd0> (a 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionSerializer.deserialize(DataSerializableSessionSerializer.java:156)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$SpyingDataSerializableSessionSerializer.deserialize(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:476)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$SpyingDataSerializableSessionSerializer.deserialize(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:441)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.fromData(AbstractDataSerializableSessionSerializer.java:107)
        at 
org.apache.geode.internal.InternalDataSerializer.readUserObject(InternalDataSerializer.java:1786)
        at 
org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2878)
        at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2978)
        at org.apache.geode.CopyHelper.doDeepCopy(CopyHelper.java:242)
        at org.apache.geode.CopyHelper.copy(CopyHelper.java:190)
        at 
org.apache.geode.internal.cache.EntryEventImpl.getNewValue(EntryEventImpl.java:1063)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$SessionEventHandlerCacheListenerAdapter$$Lambda$446/595798955.apply(Unknown
 Source)
        at java.util.Optional.map(Optional.java:215)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$SessionEventHandlerCacheListenerAdapter.isSession(AbstractGemFireOperationsSessionRepository.java:1736)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$SessionEventHandlerCacheListenerAdapter$$Lambda$445/613616595.test(Unknown
 Source)
        at java.util.Optional.filter(Optional.java:178)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$SessionEventHandlerCacheListenerAdapter.remember(AbstractGemFireOperationsSessionRepository.java:1703)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$SessionEventHandlerCacheListenerAdapter$$Lambda$444/138059244.test(Unknown
 Source)
        at java.util.Optional.filter(Optional.java:178)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$SessionEventHandlerCacheListenerAdapter.afterCreate(AbstractGemFireOperationsSessionRepository.java:1485)
        at 
org.apache.geode.internal.cache.EnumListenerEvent$AFTER_CREATE.dispatchEvent(EnumListenerEvent.java:130)
        at 
org.apache.geode.internal.cache.LocalRegion.dispatchEvent(LocalRegion.java:8485)
        at 
org.apache.geode.internal.cache.LocalRegion.dispatchListenerEvent(LocalRegion.java:7049)
        at 
org.apache.geode.internal.cache.LocalRegion.invokePutCallbacks(LocalRegion.java:6013)
        at 
org.apache.geode.internal.cache.EntryEventImpl.invokeCallbacks(EntryEventImpl.java:2402)
        at 
org.apache.geode.internal.cache.ProxyRegionMap$ProxyRegionEntry.dispatchListenerEvents(ProxyRegionMap.java:615)
        at 
org.apache.geode.internal.cache.LocalRegion.basicPutPart2(LocalRegion.java:5869)
        at 
org.apache.geode.internal.cache.ProxyRegionMap.basicPut(ProxyRegionMap.java:244)
        at 
org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5691)
        at 
org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:162)
        at 
org.apache.geode.internal.cache.LocalRegion.basicPut(LocalRegion.java:5119)
        at 
org.apache.geode.internal.cache.LocalRegion.validatedPut(LocalRegion.java:1661)
        at 
org.apache.geode.internal.cache.LocalRegion.put(LocalRegion.java:1648)
        at 
org.apache.geode.internal.cache.AbstractRegion.put(AbstractRegion.java:421)
        at 
org.springframework.data.gemfire.GemfireTemplate.put(GemfireTemplate.java:194)
        at 
org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.doSave(GemFireOperationsSessionRepository.java:226)
        at 
org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.save(GemFireOperationsSessionRepository.java:186)
        at 
org.springframework.session.data.gemfire.AbstractGemFireIntegrationTests.save(AbstractGemFireIntegrationTests.java:409)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.lambda$newAddSessionAttributeTask$2(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:238)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$$Lambda$474/775341308.call(Unknown
 Source)
{noformat}
For the BLOCKED threads:

GemFireSession.hasDelta is synchronized on the DeltaCapableGemFireSession 
instance:
{noformat}
public synchronized boolean hasDelta() {...}
{noformat}
That means one Session Thread retrieved the DeltaCapableGemFireSession from the 
server, and the above 86 Session Threads are deepCopying that same instance.

For the waiting threads:

These threads are waiting for another Session Thread to get the 
DeltaCapableGemFireSession instance from the server.

So, that is 110 Session Threads that don't have a DeltaCapableGemFireSession 
instance. That leaves only 70 Session Threads that could have a 
DeltaCapableGemFireSession instance.

master:

A thread dump shows 180 Session Threads.

93 Session Threads are serializing a DeltaCapableGemFireSession in 
CopyHelper.copy in SessionEventHandlerCacheListenerAdapter.afterCreate here:
{noformat}
"Session Thread 164" #199 daemon prio=5 os_prio=31 tid=0x00007f89d184c800 
nid=0x18a03 runnable [0x0000700016a3c000]
   java.lang.Thread.State: RUNNABLE
        at 
java.util.concurrent.CopyOnWriteArrayList.iterator(CopyOnWriteArrayList.java:1076)
        at org.slf4j.helpers.BasicMarker.contains(BasicMarker.java:99)
        at org.apache.logging.slf4j.SLF4JLogger.getMarker(SLF4JLogger.java:101)
        at 
org.apache.logging.slf4j.SLF4JLogger.isEnabledFor(SLF4JLogger.java:208)
        at org.apache.logging.slf4j.SLF4JLogger.isEnabled(SLF4JLogger.java:121)
        at 
org.apache.logging.log4j.spi.ExtendedLoggerWrapper.isEnabled(ExtendedLoggerWrapper.java:93)
        at 
org.apache.logging.log4j.spi.AbstractLogger.isTraceEnabled(AbstractLogger.java:1550)
        at 
org.apache.geode.logging.internal.log4j.api.FastLogger.isTraceEnabled(FastLogger.java:69)
        at org.apache.geode.DataSerializer.writeLong(DataSerializer.java:690)
        at 
org.apache.geode.internal.InternalDataSerializer$13.toData(InternalDataSerializer.java:567)
        at 
org.apache.geode.internal.InternalDataSerializer.writeUserObject(InternalDataSerializer.java:1564)
        at 
org.apache.geode.internal.InternalDataSerializer.writeWellKnownObject(InternalDataSerializer.java:1526)
        at 
org.apache.geode.internal.InternalDataSerializer.basicWriteObject(InternalDataSerializer.java:2105)
        at org.apache.geode.DataSerializer.writeObject(DataSerializer.java:2844)
        at org.apache.geode.DataSerializer.writeObject(DataSerializer.java:2809)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.serializeObject(AbstractDataSerializableSessionSerializer.java:102)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.serializeObject(AbstractDataSerializableSessionSerializer.java:98)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer.lambda$null$2(DataSerializableSessionAttributesSerializer.java:98)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer$$Lambda$556/2134155588.doWrite(Unknown
 Source)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.safeWrite(AbstractDataSerializableSessionSerializer.java:136)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer.lambda$serialize$3(DataSerializableSessionAttributesSerializer.java:98)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer$$Lambda$492/2054410514.accept(Unknown
 Source)
        at java.util.HashMap$KeySet.forEach(HashMap.java:932)
        at 
java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer.serialize(DataSerializableSessionAttributesSerializer.java:96)
        - locked <0x0000000776ddfb50> (a 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer.serialize(DataSerializableSessionAttributesSerializer.java:50)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.lambda$toData$0(AbstractDataSerializableSessionSerializer.java:91)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer$$Lambda$484/92577249.apply(Unknown
 Source)
        at java.util.Optional.map(Optional.java:215)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.toData(AbstractDataSerializableSessionSerializer.java:90)
        at 
org.apache.geode.internal.InternalDataSerializer.writeUserObject(InternalDataSerializer.java:1564)
        at 
org.apache.geode.internal.InternalDataSerializer.writeWellKnownObject(InternalDataSerializer.java:1526)
        at 
org.apache.geode.internal.InternalDataSerializer.basicWriteObject(InternalDataSerializer.java:2105)
        at org.apache.geode.DataSerializer.writeObject(DataSerializer.java:2844)
        at org.apache.geode.DataSerializer.writeObject(DataSerializer.java:2809)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.serializeObject(AbstractDataSerializableSessionSerializer.java:102)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.serializeObject(AbstractDataSerializableSessionSerializer.java:98)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionSerializer.lambda$serialize$6(DataSerializableSessionSerializer.java:115)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionSerializer$$Lambda$490/646457989.doWrite(Unknown
 Source)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.safeWrite(AbstractDataSerializableSessionSerializer.java:136)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionSerializer.serialize(DataSerializableSessionSerializer.java:115)
        - locked <0x0000000776ddfb50> (a 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$SpyingDataSerializableSessionSerializer.serialize(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:468)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$SpyingDataSerializableSessionSerializer.serialize(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:439)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.lambda$toData$0(AbstractDataSerializableSessionSerializer.java:91)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer$$Lambda$484/92577249.apply(Unknown
 Source)
        at java.util.Optional.map(Optional.java:215)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.toData(AbstractDataSerializableSessionSerializer.java:90)
        at 
org.apache.geode.internal.InternalDataSerializer.writeUserObject(InternalDataSerializer.java:1564)
        at 
org.apache.geode.internal.InternalDataSerializer.writeWellKnownObject(InternalDataSerializer.java:1526)
        at 
org.apache.geode.internal.InternalDataSerializer.basicWriteObject(InternalDataSerializer.java:2105)
        at org.apache.geode.DataSerializer.writeObject(DataSerializer.java:2844)
        at org.apache.geode.CopyHelper.doDeepCopy(CopyHelper.java:241)
        at org.apache.geode.CopyHelper.copy(CopyHelper.java:190)
        at 
org.apache.geode.internal.cache.EntryEventImpl.getNewValue(EntryEventImpl.java:1089)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$SessionEventHandlerCacheListenerAdapter$$Lambda$495/1861704415.apply(Unknown
 Source)
        at java.util.Optional.map(Optional.java:215)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$SessionEventHandlerCacheListenerAdapter.isSession(AbstractGemFireOperationsSessionRepository.java:1738)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$SessionEventHandlerCacheListenerAdapter$$Lambda$494/660734580.test(Unknown
 Source)
        at java.util.Optional.filter(Optional.java:178)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$SessionEventHandlerCacheListenerAdapter.remember(AbstractGemFireOperationsSessionRepository.java:1705)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$SessionEventHandlerCacheListenerAdapter$$Lambda$493/1140699224.test(Unknown
 Source)
        at java.util.Optional.filter(Optional.java:178)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$SessionEventHandlerCacheListenerAdapter.afterCreate(AbstractGemFireOperationsSessionRepository.java:1486)
        at 
org.apache.geode.internal.cache.EnumListenerEvent$AFTER_CREATE.dispatchEvent(EnumListenerEvent.java:130)
        at 
org.apache.geode.internal.cache.LocalRegion.dispatchEvent(LocalRegion.java:8214)
        at 
org.apache.geode.internal.cache.LocalRegion.dispatchListenerEvent(LocalRegion.java:6924)
        at 
org.apache.geode.internal.cache.LocalRegion.invokePutCallbacks(LocalRegion.java:5898)
        at 
org.apache.geode.internal.cache.EntryEventImpl.invokeCallbacks(EntryEventImpl.java:2440)
        at 
org.apache.geode.internal.cache.ProxyRegionMap$ProxyRegionEntry.dispatchListenerEvents(ProxyRegionMap.java:615)
        at 
org.apache.geode.internal.cache.LocalRegion.basicPutPart2(LocalRegion.java:5756)
        at 
org.apache.geode.internal.cache.ProxyRegionMap.basicPut(ProxyRegionMap.java:244)
        at 
org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5580)
        at 
org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:162)
        at 
org.apache.geode.internal.cache.LocalRegion.basicPut(LocalRegion.java:5036)
        at 
org.apache.geode.internal.cache.LocalRegion.validatedPut(LocalRegion.java:1635)
        at 
org.apache.geode.internal.cache.LocalRegion.put(LocalRegion.java:1622)
        at 
org.apache.geode.internal.cache.AbstractRegion.put(AbstractRegion.java:442)
        at 
org.springframework.data.gemfire.GemfireTemplate.put(GemfireTemplate.java:197)
        at 
org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.doSave(GemFireOperationsSessionRepository.java:226)
        at 
org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.save(GemFireOperationsSessionRepository.java:186)
        at 
org.springframework.session.data.gemfire.AbstractGemFireIntegrationTests.save(AbstractGemFireIntegrationTests.java:409)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.lambda$newAddSessionAttributeTask$2(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:239)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$$Lambda$523/1237205245.call(Unknown
 Source)
{noformat}
54 Session Threads copying a DeltaCapableGemFireSession in CopyHelper.copy 
during a get here:
{noformat}
"Session Thread 179" #214 daemon prio=5 os_prio=31 tid=0x00007f89cf0a2000 
nid=0x19903 runnable [0x0000700017969000]
   java.lang.Thread.State: RUNNABLE
        at org.slf4j.helpers.BasicMarker.contains(BasicMarker.java:103)
        at org.apache.logging.slf4j.SLF4JLogger.getMarker(SLF4JLogger.java:101)
        at 
org.apache.logging.slf4j.SLF4JLogger.isEnabledFor(SLF4JLogger.java:208)
        at org.apache.logging.slf4j.SLF4JLogger.isEnabled(SLF4JLogger.java:121)
        at 
org.apache.logging.log4j.spi.ExtendedLoggerWrapper.isEnabled(ExtendedLoggerWrapper.java:93)
        at 
org.apache.logging.log4j.spi.AbstractLogger.isTraceEnabled(AbstractLogger.java:1550)
        at 
org.apache.geode.logging.internal.log4j.api.FastLogger.isTraceEnabled(FastLogger.java:69)
        at org.apache.geode.DataSerializer.readLong(DataSerializer.java:706)
        at 
org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2667)
        at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2882)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.deserializeObject(AbstractDataSerializableSessionSerializer.java:111)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer$$Lambda$552/721863113.doRead(Unknown
 Source)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.safeRead(AbstractDataSerializableSessionSerializer.java:126)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer.deserialize(DataSerializableSessionAttributesSerializer.java:109)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer.deserialize(DataSerializableSessionAttributesSerializer.java:50)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.fromData(AbstractDataSerializableSessionSerializer.java:107)
        at 
org.apache.geode.internal.InternalDataSerializer.readUserObject(InternalDataSerializer.java:1724)
        at 
org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2719)
        at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2882)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.deserializeObject(AbstractDataSerializableSessionSerializer.java:111)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionSerializer$$Lambda$502/1894017486.doRead(Unknown
 Source)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.safeRead(AbstractDataSerializableSessionSerializer.java:126)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionSerializer.deserialize(DataSerializableSessionSerializer.java:156)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$SpyingDataSerializableSessionSerializer.deserialize(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:475)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$SpyingDataSerializableSessionSerializer.deserialize(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:439)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.fromData(AbstractDataSerializableSessionSerializer.java:107)
        at 
org.apache.geode.internal.InternalDataSerializer.readUserObject(InternalDataSerializer.java:1724)
        at 
org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2719)
        at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2882)
        at org.apache.geode.CopyHelper.doDeepCopy(CopyHelper.java:242)
        at org.apache.geode.CopyHelper.copy(CopyHelper.java:190)
        at 
org.apache.geode.internal.cache.LocalRegion.conditionalCopy(LocalRegion.java:1581)
        at 
org.apache.geode.internal.cache.LocalRegion.getObject(LocalRegion.java:1488)
        at 
org.apache.geode.internal.cache.LocalRegion.nonTxnFindObject(LocalRegion.java:1443)
        at 
org.apache.geode.internal.cache.LocalRegionDataView.findObject(LocalRegionDataView.java:188)
        at 
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1380)
        at 
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1319)
        at 
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1304)
        at 
org.apache.geode.internal.cache.AbstractRegion.get(AbstractRegion.java:436)
        at 
org.springframework.data.gemfire.GemfireTemplate.get(GemfireTemplate.java:169)
        at 
org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.findById(GemFireOperationsSessionRepository.java:95)
        at 
org.springframework.session.data.gemfire.AbstractGemFireIntegrationTests.get(AbstractGemFireIntegrationTests.java:405)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.lambda$newAddSessionAttributeTask$2(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:221)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$$Lambda$523/1237205245.call(Unknown
 Source)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
{noformat}
32 Session Threads creating a DeltaCapableGemFireSession from a response from 
the server:
{noformat}
"Session Thread 9" #44 daemon prio=5 os_prio=31 tid=0x00007f89ce144000 
nid=0x8503 runnable [0x000070000cd6b000]
   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.resize(HashMap.java:734)
        at java.util.HashMap.putVal(HashMap.java:662)
        at java.util.HashMap.put(HashMap.java:611)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes.doSetAttribute(AbstractGemFireOperationsSessionRepository.java:1322)
        at 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes.setAttribute(AbstractGemFireOperationsSessionRepository.java:1313)
        - locked <0x0000000779ca9a08> (a 
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer.deserialize(DataSerializableSessionAttributesSerializer.java:109)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer.deserialize(DataSerializableSessionAttributesSerializer.java:50)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.fromData(AbstractDataSerializableSessionSerializer.java:107)
        at 
org.apache.geode.internal.InternalDataSerializer.readUserObject(InternalDataSerializer.java:1724)
        at 
org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2719)
        at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2882)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.deserializeObject(AbstractDataSerializableSessionSerializer.java:111)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionSerializer$$Lambda$502/1894017486.doRead(Unknown
 Source)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.safeRead(AbstractDataSerializableSessionSerializer.java:126)
        at 
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionSerializer.deserialize(DataSerializableSessionSerializer.java:156)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$SpyingDataSerializableSessionSerializer.deserialize(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:475)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$SpyingDataSerializableSessionSerializer.deserialize(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:439)
        at 
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.fromData(AbstractDataSerializableSessionSerializer.java:107)
        at 
org.apache.geode.internal.InternalDataSerializer.readUserObject(InternalDataSerializer.java:1724)
        at 
org.apache.geode.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2719)
        at org.apache.geode.DataSerializer.readObject(DataSerializer.java:2882)
        at 
org.apache.geode.internal.util.BlobHelper.deserializeBlob(BlobHelper.java:99)
        at 
org.apache.geode.internal.cache.tier.sockets.CacheServerHelper.deserialize(CacheServerHelper.java:73)
        at 
org.apache.geode.internal.cache.tier.sockets.Part.getObject(Part.java:351)
        at 
org.apache.geode.internal.cache.tier.sockets.Part.getObject(Part.java:359)
        at 
org.apache.geode.cache.client.internal.AbstractOp.processObjResponse(AbstractOp.java:282)
        at 
org.apache.geode.cache.client.internal.GetOp$GetOpImpl.processResponse(GetOp.java:143)
        at 
org.apache.geode.cache.client.internal.AbstractOp.attemptReadResponse(AbstractOp.java:207)
        at 
org.apache.geode.cache.client.internal.AbstractOp.attempt(AbstractOp.java:382)
        at 
org.apache.geode.cache.client.internal.ConnectionImpl.execute(ConnectionImpl.java:263)
        - locked <0x0000000740d30be8> (a 
org.apache.geode.cache.client.internal.ConnectionImpl)
        at 
org.apache.geode.cache.client.internal.pooling.PooledConnection.execute(PooledConnection.java:353)
        at 
org.apache.geode.cache.client.internal.OpExecutorImpl.executeWithPossibleReAuthentication(OpExecutorImpl.java:750)
        at 
org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:138)
        at 
org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:108)
        at 
org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:772)
        at org.apache.geode.cache.client.internal.GetOp.execute(GetOp.java:91)
        at 
org.apache.geode.cache.client.internal.ServerRegionProxy.get(ServerRegionProxy.java:116)
        at 
org.apache.geode.internal.cache.LocalRegion.findObjectInSystem(LocalRegion.java:2793)
        at 
org.apache.geode.internal.cache.LocalRegion.getObject(LocalRegion.java:1470)
        at 
org.apache.geode.internal.cache.LocalRegion.nonTxnFindObject(LocalRegion.java:1443)
        at 
org.apache.geode.internal.cache.LocalRegionDataView.findObject(LocalRegionDataView.java:188)
        at 
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1380)
        at 
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1319)
        at 
org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1304)
        at 
org.apache.geode.internal.cache.AbstractRegion.get(AbstractRegion.java:436)
        at 
org.springframework.data.gemfire.GemfireTemplate.get(GemfireTemplate.java:169)
        at 
org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.findById(GemFireOperationsSessionRepository.java:95)
        at 
org.springframework.session.data.gemfire.AbstractGemFireIntegrationTests.get(AbstractGemFireIntegrationTests.java:405)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.lambda$newAddSessionAttributeTask$2(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:221)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$$Lambda$523/1237205245.call(Unknown
 Source)
{noformat}
1 Session Thread waiting for the response from the server during a put:
{noformat}
"Session Thread 111" #146 daemon prio=5 os_prio=31 tid=0x00007f89cf093000 
nid=0x15a03 runnable [0x000070001349d000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at 
org.apache.geode.internal.cache.tier.sockets.Message.fetchHeader(Message.java:849)
        at 
org.apache.geode.internal.cache.tier.sockets.Message.readHeaderAndBody(Message.java:696)
        at 
org.apache.geode.internal.cache.tier.sockets.Message.receive(Message.java:1172)
        - locked <0x00000007599770d8> (a java.nio.HeapByteBuffer)
        at 
org.apache.geode.cache.client.internal.AbstractOp.attemptReadResponse(AbstractOp.java:202)
        at 
org.apache.geode.cache.client.internal.AbstractOp.attempt(AbstractOp.java:382)
        at 
org.apache.geode.cache.client.internal.ConnectionImpl.execute(ConnectionImpl.java:263)
        - locked <0x0000000759976eb8> (a 
org.apache.geode.cache.client.internal.ConnectionImpl)
        at 
org.apache.geode.cache.client.internal.pooling.PooledConnection.execute(PooledConnection.java:353)
        at 
org.apache.geode.cache.client.internal.OpExecutorImpl.executeWithPossibleReAuthentication(OpExecutorImpl.java:750)
        at 
org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:138)
        at 
org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:108)
        at 
org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:772)
        at org.apache.geode.cache.client.internal.PutOp.execute(PutOp.java:89)
        at 
org.apache.geode.cache.client.internal.ServerRegionProxy.put(ServerRegionProxy.java:159)
        at 
org.apache.geode.internal.cache.LocalRegion.serverPut(LocalRegion.java:3035)
        at 
org.apache.geode.internal.cache.LocalRegion.cacheWriteBeforePut(LocalRegion.java:3152)
        at 
org.apache.geode.internal.cache.ProxyRegionMap.basicPut(ProxyRegionMap.java:238)
        at 
org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5580)
        at 
org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:162)
        at 
org.apache.geode.internal.cache.LocalRegion.basicPut(LocalRegion.java:5036)
        at 
org.apache.geode.internal.cache.LocalRegion.validatedPut(LocalRegion.java:1635)
        at 
org.apache.geode.internal.cache.LocalRegion.put(LocalRegion.java:1622)
        at 
org.apache.geode.internal.cache.AbstractRegion.put(AbstractRegion.java:442)
        at 
org.springframework.data.gemfire.GemfireTemplate.put(GemfireTemplate.java:197)
        at 
org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.doSave(GemFireOperationsSessionRepository.java:226)
        at 
org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.save(GemFireOperationsSessionRepository.java:186)
        at 
org.springframework.session.data.gemfire.AbstractGemFireIntegrationTests.save(AbstractGemFireIntegrationTests.java:409)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.lambda$newAddSessionAttributeTask$2(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:239)
        at 
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$$Lambda$523/1237205245.call(Unknown
 Source)
{noformat}
Each of these threads has 1 or more actual DeltaCapableGemFireSession instances 
or the bytes for one.

That helps explain why there are far fewer DeltaCapableGemFireSession instances 
in 2.2.2.RELEASE than in master and why there is additional memory usage in 
master.


> Apache Geode 1.11 severely and negatively impacts performance and resource 
> utilization
> --------------------------------------------------------------------------------------
>
>                 Key: GEODE-7763
>                 URL: https://issues.apache.org/jira/browse/GEODE-7763
>             Project: Geode
>          Issue Type: Bug
>          Components: client/server
>    Affects Versions: 1.10.0, 1.11.0
>            Reporter: John Blum
>            Priority: Critical
>              Labels: performance
>         Attachments: 1.11-client-stats.gfs, 1.11-server-stats.gfs, 
> 1.11_thread_dumps.rtf, 1.9-client-stats.gfs, 1.9-server-stats.gfs, 1.9.log, 
> apache-geode-1.10-client-server-interaction-output.txt, 
> apache-geode-1.10-client-server-startup-output.txt, 
> apache-geode-1.11-client-server-interaction-output.txt, 
> apache-geode-1.11-client-server-startup-output.txt
>
>
> This problem was first observed in Apache Geode 1.11.0.  The problem was not 
> present in Apache Geode 1.9.2.  This problem is an issue for Apache Geode 
> 1.10 as well!
> After upgrading _Spring Session for Apache Geode_ (SSDG) 2.3 to _Spring Data 
> for Apache Geode_ (SDG) Neumann/2.3, which is based on Apache Geode 1.11, 
> this problem with SSDG's test suite started occurring.
>  _Spring Session for Apache Geode_ (SSDG) 2.2, which is based on _Spring Data 
> for Apache Geode_ (SDG) Moore/2.2, pulls in Apache Geode 1.9.2.  This problem 
> did not occur in SSDG 2.2. with Apache Geode 1.9.2.
> Out of curiosity, I wondered whether this problem affects (i.e. was actually 
> introduced in) Apache Geode 1.10.0.  So, I configured SSDG 2.3 to pull in SDG 
> Moore/2.2 but run with Apache Geode 1.10. The problem occurred with Apache 
> Geode 1.10 as well!
> The SSDG test class in question, affected by Geode's deficiencies, is the 
> [MultiThreadedHighlyConcurrentClientServerSessionOperationsIntegrationTests|https://github.com/spring-projects/spring-session-data-geode/blob/2.2.2.RELEASE/spring-session-data-geode/src/integration-test/java/org/springframework/session/data/gemfire/MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java].
> The test class was modeled after a customer UC, who were using Spring Session 
> and Apache Geode/Pivotal GemFire as the HTTP Session state management 
> provider, therefore it simulates their highly concurrent environment.
> The test class has 2 primary parameters: [Thread 
> Count|https://github.com/spring-projects/spring-session-data-geode/blob/2.2.2.RELEASE/spring-session-data-geode/src/integration-test/java/org/springframework/session/data/gemfire/MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java#L90]
>  and the [Workload 
> Size|https://github.com/spring-projects/spring-session-data-geode/blob/2.2.2.RELEASE/spring-session-data-geode/src/integration-test/java/org/springframework/session/data/gemfire/MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java#L91].
> The "_Workload Size_" should not be confused with the "_Payload Size_" of the 
> individual objects passed to the Geode data access operations (i.e. {{gets}}, 
> {{puts}}, {{removes}}).  The "_Workload Size_" merely determines the number 
> of {{get}}, {{put}} or {{remove}} operations performed on the (Session) 
> Region over the duration of the test run.  Certain operations are "favored" 
> over others, therefore the number of {{gets}}, {{puts}} and {{removes}} is 
> weighted.
> The "_Payload_" in this case is a (HTTP) {{Session}} object and the "size" is 
> directly proportional to the number of Session attributes stored in the 
> Session.
> As you can see from the [test class 
> configuration|https://github.com/spring-projects/spring-session-data-geode/blob/2.2.2.RELEASE/spring-session-data-geode/src/integration-test/java/org/springframework/session/data/gemfire/MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java#L90-L91]
>  in *SSDG* {{2.2}}, the *Thread Count* was set to *180* and the *Workload 
> Size* (or number of Region operations) was set to *10,000*.
> This had to be significantly adjusted in SSDG 2.3 using Apache Geode 1.11 
> (and, as it turns out, Apache Geode 1.10 as well), as can be seen in the 
> {{2.3.0.M1}} release bits source, 
> [here|https://github.com/spring-projects/spring-session-data-geode/blob/2.3.0.M1/spring-session-data-geode/src/integration-test/java/org/springframework/session/data/gemfire/MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java#L94-L95].
> It turns out different combinations of the Thread Count (number of workers, 
> or "concurrent Sessions") and Workload Size ultimately determine whether this 
> test class passes or not.
> In other words, if I increase the Thread Count, then the Workload Size must 
> decrease, otherwise the test fails!  If I increase the Workload Size, then 
> the Thread Count must decrease, otherwise again the test fails!
> I tried with different combinations of Thread Count and Workload Size until 
> the test passed.  More often than not 180 Threads with 3000 Regions 
> operations worked, but was right on the cusp of failing, therefore, I settled 
> on 180 Threads (which nearly matches the customers environment of 200 
> concurrent client Sessions) and 2000 concurrent Region operations.
> The point of the test class is to assert the state of the Session is 
> consistent at the end of the test run.
> However, before this test can even finish, the client, as in the 
> {{ClientCache}} instance, starts failing with Exceptions, specifically:
> {code:java}
> java.lang.RuntimeException: Session Access Task Failed
>       at 
> org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.safeFutureGet(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:298)
>       at 
> java.util.stream.ReferencePipeline$4$1.accept(ReferencePipeline.java:210)
>       at 
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
>       at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
>       at 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
>       at 
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
>       at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
>       at java.util.stream.IntPipeline.reduce(IntPipeline.java:456)
>       at java.util.stream.IntPipeline.sum(IntPipeline.java:414)
>       at 
> org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.runSessionWorkload(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:313)
>       at 
> org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.concurrentSessionAccessIsCorrect(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:324)
>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>       at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>       at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>       at java.lang.reflect.Method.invoke(Method.java:498)
>       at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>       at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>       at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>       at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>       at 
> org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:74)
>       at 
> org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:84)
>       at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>       at 
> org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
>       at 
> org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
>       at 
> org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
>       at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>       at 
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251)
>       at 
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
>       at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>       at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>       at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>       at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>       at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>       at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>       at 
> org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
>       at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>       at 
> org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
>       at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>       at 
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
>       at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>       at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
>       at 
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
>       at 
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
>       at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
> Caused by: java.util.concurrent.ExecutionException: 
> org.springframework.dao.DataAccessResourceFailureException: Pool unexpected 
> socket timed out on client connection=Pooled Connection to localhost:60964: 
> Connection[DESTROYED]). Server unreachable: could not connect after 1 
> attempts; nested exception is 
> org.apache.geode.cache.client.ServerConnectivityException: Pool unexpected 
> socket timed out on client connection=Pooled Connection to localhost:60964: 
> Connection[DESTROYED]). Server unreachable: could not connect after 1 attempts
>       at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>       at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>       at 
> org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.safeFutureGet(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:295)
>       ... 43 more
> Caused by: org.springframework.dao.DataAccessResourceFailureException: Pool 
> unexpected socket timed out on client connection=Pooled Connection to 
> localhost:60964: Connection[DESTROYED]). Server unreachable: could not 
> connect after 1 attempts; nested exception is 
> org.apache.geode.cache.client.ServerConnectivityException: Pool unexpected 
> socket timed out on client connection=Pooled Connection to localhost:60964: 
> Connection[DESTROYED]). Server unreachable: could not connect after 1 attempts
>       at 
> org.springframework.data.gemfire.GemfireCacheUtils.convertGemfireAccessException(GemfireCacheUtils.java:235)
>       at 
> org.springframework.data.gemfire.GemfireAccessor.convertGemFireAccessException(GemfireAccessor.java:93)
>       at 
> org.springframework.data.gemfire.GemfireTemplate.put(GemfireTemplate.java:200)
>       at 
> org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.doSave(GemFireOperationsSessionRepository.java:226)
>       at 
> org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.save(GemFireOperationsSessionRepository.java:186)
>       at 
> org.springframework.session.data.gemfire.AbstractGemFireIntegrationTests.save(AbstractGemFireIntegrationTests.java:409)
>       at 
> org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.lambda$newAddSessionAttributeTask$2(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:216)
>       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>       at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>       at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>       at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.geode.cache.client.ServerConnectivityException: Pool 
> unexpected socket timed out on client connection=Pooled Connection to 
> localhost:60964: Connection[DESTROYED]). Server unreachable: could not 
> connect after 1 attempts
>       at 
> org.apache.geode.cache.client.internal.OpExecutorImpl.handleException(OpExecutorImpl.java:659)
>       at 
> org.apache.geode.cache.client.internal.OpExecutorImpl.handleException(OpExecutorImpl.java:501)
>       at 
> org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:153)
>       at 
> org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:108)
>       at 
> org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:772)
>       at org.apache.geode.cache.client.internal.PutOp.execute(PutOp.java:89)
>       at 
> org.apache.geode.cache.client.internal.ServerRegionProxy.put(ServerRegionProxy.java:159)
>       at 
> org.apache.geode.internal.cache.LocalRegion.serverPut(LocalRegion.java:3035)
>       at 
> org.apache.geode.internal.cache.LocalRegion.cacheWriteBeforePut(LocalRegion.java:3152)
>       at 
> org.apache.geode.internal.cache.ProxyRegionMap.basicPut(ProxyRegionMap.java:238)
>       at 
> org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5580)
>       at 
> org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:162)
>       at 
> org.apache.geode.internal.cache.LocalRegion.basicPut(LocalRegion.java:5036)
>       at 
> org.apache.geode.internal.cache.LocalRegion.validatedPut(LocalRegion.java:1635)
>       at 
> org.apache.geode.internal.cache.LocalRegion.put(LocalRegion.java:1622)
>       at 
> org.apache.geode.internal.cache.AbstractRegion.put(AbstractRegion.java:442)
>       at 
> org.springframework.data.gemfire.GemfireTemplate.put(GemfireTemplate.java:197)
>       ... 8 more
> {code}
> Attached to this issue are log output files from each of my runs using Apache 
> Geode 1.10 and 1.11.
> The log files serve 2 purposes: 1) to show the version of Apache Geode used 
> and 2) the errors occurs on the client and server during the run.
> Any lines in the log output prefixed with "{{[FORK]}}" originates from the 
> cache server.  The other lines come from the client.  There is only a single 
> client and server in this test case.
> It takes a bit of initial time during the run for the failures to start 
> occurring, which is why this seems like a resource utilization problem.
> After first, I suspected issues with the client Pool configuration, or 
> {{CacheServer}} configuration, adjusting timeouts and so on.  I even 
> suspected memory being an issue for the client and server processes, upping 
> each to 2 GB+.  However, it turns out none of the changes made a bit of 
> difference.  And the truth of the matter is, this (existing) configuration 
> worked seamlessly until I upgraded to Apache Geode 1.10+ (specifically, 1.11).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to