> On March 17, 2016, 5:06 p.m., Dan Smith wrote:
> > What are the reasons why the serialized value is not available? Could this
> > change cause us to drop events without realizing it due to bugs in getting
> > the off heap value?
The data PR is locally destroyed which in turn locally destroys the queue PR.
This clears the AbstractRegionMap which iterates the entries and releases them:
[warn 2016/03/17 12:37:45.423 PDT <RMI TCP Connection(1)-192.168.2.7> tid=0x13]
AAA GatewaySenderEventImpl.release released key: 5
java.lang.Exception
at
com.gemstone.gemfire.internal.cache.wan.GatewaySenderEventImpl.release(GatewaySenderEventImpl.java:1259)
at
com.gemstone.gemfire.internal.cache.wan.GatewaySenderEventImpl.release(GatewaySenderEventImpl.java:1265)
at
com.gemstone.gemfire.internal.util.concurrent.CustomEntryConcurrentHashMap$Segment.clear(CustomEntryConcurrentHashMap.java:1114)
at
com.gemstone.gemfire.internal.util.concurrent.CustomEntryConcurrentHashMap.clear(CustomEntryConcurrentHashMap.java:1976)
at
com.gemstone.gemfire.internal.cache.AbstractRegionMap._mapClear(AbstractRegionMap.java:421)
at
com.gemstone.gemfire.internal.cache.AbstractRegionMap.clear(AbstractRegionMap.java:467)
at
com.gemstone.gemfire.internal.cache.AbstractLRURegionMap.clear(AbstractLRURegionMap.java:678)
at
com.gemstone.gemfire.internal.cache.AbstractRegionMap.close(AbstractRegionMap.java:432)
at
com.gemstone.gemfire.internal.cache.LocalRegion.closeEntries(LocalRegion.java:2979)
at
com.gemstone.gemfire.internal.cache.AbstractBucketRegionQueue.access$001(AbstractBucketRegionQueue.java:52)
at
com.gemstone.gemfire.internal.cache.AbstractBucketRegionQueue$1.run(AbstractBucketRegionQueue.java:503)
at
com.gemstone.gemfire.internal.offheap.OffHeapRegionEntryHelper.doWithOffHeapClear(OffHeapRegionEntryHelper.java:409)
at
com.gemstone.gemfire.internal.cache.AbstractBucketRegionQueue.closeEntries(AbstractBucketRegionQueue.java:500)
at
com.gemstone.gemfire.internal.cache.BucketRegionQueue.access$001(BucketRegionQueue.java:61)
at
com.gemstone.gemfire.internal.cache.BucketRegionQueue$2.run(BucketRegionQueue.java:227)
at
com.gemstone.gemfire.internal.offheap.OffHeapRegionEntryHelper.doWithOffHeapClear(OffHeapRegionEntryHelper.java:409)
at
com.gemstone.gemfire.internal.cache.BucketRegionQueue.closeEntries(BucketRegionQueue.java:224)
at
com.gemstone.gemfire.internal.cache.DiskStoreImpl.basicDestroy(DiskStoreImpl.java:2673)
at
com.gemstone.gemfire.internal.cache.DiskStoreImpl.endDestroyRegion(DiskStoreImpl.java:3273)
at
com.gemstone.gemfire.internal.cache.AbstractDiskRegion.endDestroy(AbstractDiskRegion.java:564)
at
com.gemstone.gemfire.internal.cache.AbstractDiskRegion.beginDestroy(AbstractDiskRegion.java:581)
at
com.gemstone.gemfire.internal.cache.PartitionedRegion.closePartitionedRegion(PartitionedRegion.java:7694)
at
com.gemstone.gemfire.internal.cache.PartitionedRegion.postDestroyRegion(PartitionedRegion.java:8362)
at
com.gemstone.gemfire.internal.cache.LocalRegion.recursiveDestroyRegion(LocalRegion.java:2948)
at
com.gemstone.gemfire.internal.cache.LocalRegion.basicDestroyRegion(LocalRegion.java:6886)
at
com.gemstone.gemfire.internal.cache.PartitionedRegion.destroyParallelGatewaySenderRegion(PartitionedRegion.java:7901)
at
com.gemstone.gemfire.internal.cache.LocalRegion.basicDestroyRegion(LocalRegion.java:6871)
at
com.gemstone.gemfire.internal.cache.LocalRegion.basicDestroyRegion(LocalRegion.java:6834)
at
com.gemstone.gemfire.internal.cache.PartitionedRegion.localDestroyRegion(PartitionedRegion.java:7984)
at
com.gemstone.gemfire.internal.cache.PartitionedRegion.localDestroyRegion(PartitionedRegion.java:7945)
at
com.gemstone.gemfire.internal.cache.AbstractRegion.localDestroyRegion(AbstractRegion.java:354)
at
com.gemstone.gemfire.internal.cache.wan.WANTestBase.localDestroyRegion(WANTestBase.java:3212)
There are a few things that could be looked at:
After a GatewaySenderEventImpl is peeked from the queue in
ParallelGatewaySenderQueue.peek, makeHeapCopyIfOffHeap is invoked on it. That
call doesn't really do anything.
Basically the test in GatewaySenderEventImpl.makeHeapCopyIfOffHeap (below) is
false because hasRefCount is false so 'this' is returned (no copy is made).
Should hasRefCount return true here? If it did return true, then a copy would
be made, and I think this issue would be resolved.
if (v instanceof StoredObject && ((StoredObject) v).hasRefCount()) {
Also, the queue PR is being destroyed with cacheWrite=false (set in
PartitionedRegion.localDestroyRegion):
[warn 2016/03/17 12:31:38.057 PDT <RMI TCP Connection(1)-192.168.2.7> tid=0x13]
AAA PartitionedRegion.destroyParallelGatewaySenderRegion destroying queue
region /ln2_PARALLEL_GATEWAY_SENDER_QUEUE; cacheWrite: false
java.lang.Exception
at
com.gemstone.gemfire.internal.cache.PartitionedRegion.destroyParallelGatewaySenderRegion(PartitionedRegion.java:7880)
at
com.gemstone.gemfire.internal.cache.LocalRegion.basicDestroyRegion(LocalRegion.java:6871)
at
com.gemstone.gemfire.internal.cache.LocalRegion.basicDestroyRegion(LocalRegion.java:6834)
at
com.gemstone.gemfire.internal.cache.PartitionedRegion.localDestroyRegion(PartitionedRegion.java:7984)
at
com.gemstone.gemfire.internal.cache.PartitionedRegion.localDestroyRegion(PartitionedRegion.java:7945)
at
com.gemstone.gemfire.internal.cache.AbstractRegion.localDestroyRegion(AbstractRegion.java:354)
at
com.gemstone.gemfire.internal.cache.wan.WANTestBase.localDestroyRegion(WANTestBase.java:3212)
at
com.gemstone.gemfire.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOperation_2_DUnitTest.lambda$testParallelGatewaySenders_MultiplNode_UserPR_localDestroy_RecreateXXX$76295ecb$3(ConcurrentParallelGatewaySenderOperation_2_DUnitTest.java:339)
at
com.gemstone.gemfire.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOperation_2_DUnitTest$$Lambda$31/936369667.run(Unknown
Source)
If cacheWrite were true, then the destroyParallelGatewaySenderRegion would wait
for the queue to drain before destroying the queue PR. There are other uses of
cacheWrite in this method though, so I'm not sure what the consequences of
changing this to true are.
if (cacheWrite && parallelQueueRegion.size() != 0 && keepWaiting) {
continue;
}
- Barry
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44929/#review124048
-----------------------------------------------------------
On March 16, 2016, 8:52 p.m., Barry Oglesby wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44929/
> -----------------------------------------------------------
>
> (Updated March 16, 2016, 8:52 p.m.)
>
>
> Review request for geode, anilkumar gingade and Dan Smith.
>
>
> Repository: geode
>
>
> Description
> -------
>
> GEODE-1067: The dispatcher now handles IllegalStateException by retrying
> batch without the released event
>
>
> Diffs
> -----
>
>
> geode-core/src/main/java/com/gemstone/gemfire/internal/cache/wan/AbstractGatewaySenderEventProcessor.java
> 51b125a4a51cbfde3141404dae7084e0432b341b
>
> geode-core/src/main/java/com/gemstone/gemfire/internal/cache/wan/GatewaySenderEventImpl.java
> d8922f874830831fff573f8f1a6a9ed09fbcfde8
>
> geode-wan/src/main/java/com/gemstone/gemfire/internal/cache/wan/GatewaySenderEventRemoteDispatcher.java
> 22dff3dc898cfc89c3ed1752f6cb352717140263
>
> Diff: https://reviews.apache.org/r/44929/diff/
>
>
> Testing
> -------
>
> - repeatedly ran ParallelGatewaySenderOperation_2_OffHeapDUnitTest (along
> fixes for GEODE-670 and GEODE-1095)
> - running precheckin
>
>
> Thanks,
>
> Barry Oglesby
>
>
