[
https://issues.apache.org/jira/browse/GEODE-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Baker closed GEODE-2027.
--------------------------------
> Gateway events being processed by the ParallelQueueRemovalMessage in the
> secondary at the time of GII may not be removed from the queue successfully
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: GEODE-2027
> URL: https://issues.apache.org/jira/browse/GEODE-2027
> Project: Geode
> Issue Type: Bug
> Components: wan
> Reporter: Barry Oglesby
> Assignee: Barry Oglesby
> Fix For: 1.0.0-incubating
>
>
> This is the same as Trac ticket 49196.
> I see two cases that cause this issue (so far). In both cases the member is
> starting and GIIing the bucket region, and the entry in question is included
> in the GII response.
> In case one:
> - the bucket region is not found in the {{ParallelGatewaySenderQueue.put}}
> path, so the event is added to the temp queue
> - the bucket region is found in the {{ParallelQueueRemovalMessage.process}}
> path and contains the key so it is removed from it
> - the event is added back into the bucket region from the temp queue
> The event is added to the temp queue:
> {noformat}
> [warning 2016/10/20 01:25:07.715 UTC bridgegemfire_1_1_ip-10-32-109-74_25707
> <P2P message reader for
> ip-10-32-109-74(bridgegemfire_1_2_ip-10-32-109-74_25713:25713)<ec><v1>:1026
> unshared ordered uid=75 dom #1 port=57564> tid=0x82] XXX
> ParallelGatewaySenderQueue.put adding to tempQueue brq=null; key=33283;
> value=SenderEventImpl[id=EventID[threadID=0x10007|5;sequenceID=177;bucketId=7];action=2;operation=DESTROY;region=/testRegion1;key=Object_2934;value=;valueIsObject=1;numberOfParts=8;callbackArgument=GatewaySenderEventCallbackArgument
>
> [originalCallbackArg=null;originatingSenderId=1;recipientGatewayReceivers=[2]];possibleDuplicate=false;creationTime=1476926707714;shadowKey=
> 33283;timeStamp=1476926707709;acked=false;dispatched=false]
> {noformat}
> The event exists in the bucket region and is removed:
> {noformat}
> [warning 2016/10/20 01:25:10.265 UTC bridgegemfire_1_1_ip-10-32-109-74_25707
> <Pooled Message Processor 1> tid=0x63] XXX
> ParallelQueueRemovalMessage.process bucket is not initialized but contains
> key destroying from bucket
> name=/__PR/_B__sender__bridgeds__1__to__bridgeds__2__PARALLEL__GATEWAY__SENDER__QUEUE_7;
> key=33283
> {noformat}
> The event is loaded from the temp queue into the bucket region:
> {noformat}
> [warning 2016/10/20 01:25:10.432 UTC bridgegemfire_1_1_ip-10-32-109-74_25707
> <ResourceManagerRecoveryThread 0> tid=0x4f] XXX
> AbstractBucketRegionQueue.loadEventsFromTempQueue
> _B__sender__bridgeds__1__to__bridgeds__2__PARALLEL__GATEWAY__SENDER__QUEUE_7
> adding to queue key=33283 ->
> SenderEventImpl[id=EventID[threadID=0x10007|5;sequenceID=177;bucketId=7];action=2;operation=DESTROY;region=/testRegion1;key=Object_2934;value=;valueIsObject=1;numberOfParts=8;callbackArgument=GatewaySenderEventCallbackArgument
>
> [originalCallbackArg=null;originatingSenderId=1;recipientGatewayReceivers=[2]];possibleDuplicate=true;creationTime=1476926707714;shadowKey=
> 33283;timeStamp=1476926707709;acked=false;dispatched=false]
> {noformat}
> The bucket region contains the key right before it is available for use:
> {noformat}
> [warning 2016/10/20 01:25:10.790 UTC bridgegemfire_1_1_ip-10-32-109-74_25707
> <ResourceManagerRecoveryThread 0> tid=0x4f] XXX
> BucketRegionQueue.cleanUpDestroyedTokensAndMarkGIIComplete
> _B__sender__bridgeds__1__to__bridgeds__2__PARALLEL__GATEWAY__SENDER__QUEUE_7
> keySet4=[33283, 33079, 34591, 35071, 35119, 34879, 34447, 34891, 34843,
> 35083, 33295, 34387, 34339, 32743, 35131, 35095, 35059, 34603, 34471, 34567,
> 34375, 32983, 32875, 34399, 34867, 34831, 34759, 34675, 35107, 35047, 34723,
> 34495, 34507, 34459, 34699, 34735]
> {noformat}
> In case two:
> - the bucket region is not found in the {{ParallelGatewaySenderQueue.put}}
> path, so the event is added to the temp queue
> - the bucket region is found in the {{ParallelQueueRemovalMessage.process}}
> path, but doesn't contain the key so it is removed from the temp queue
> - the event is removed from the temp queue but is also in the bucket region
> The event is added to the temp queue:
> {noformat}
> [warning 2016/10/19 22:55:27.743 UTC bridgegemfire_1_1_ip-10-32-109-74_24670
> <P2P message reader for
> ip-10-32-109-74(bridgegemfire_1_3_ip-10-32-109-74_24716:24716)<ec><v1>:1028
> unshared ordered uid=62 dom #1 port=56962> tid=0x68] XXX
> ParallelGatewaySenderQueue.put adding to tempQueue brq=null; key=31555;
> value=SenderEventImpl[id=EventID[threadID=0x10007|7;sequenceID=108;bucketId=7];action=0;operation=CREATE;region=/testRegion1;key=Object_10236;value=util.ValueHolder
> [myVersion=util.ValueHolder, myValue=<java.lang.Long, value 10236>,
> extraObject=<[B Size 70, componentType byte>,
> modVal=null];valueIsObject=1;numberOfParts=9;callbackArgument=GatewaySenderEventCallbackArgument
>
> [originalCallbackArg=July;originatingSenderId=1;recipientGatewayReceivers=[2]];possibleDuplicate=false;creationTime=1476917727742;shadowKey=
> 31555;timeStamp=1476917727723;acked=false;dispatched=false]
> {noformat}
> The event doesn't exist in the bucket region and is removed from the temp
> queue:
> {noformat}
> [warning 2016/10/19 22:55:30.047 UTC bridgegemfire_1_1_ip-10-32-109-74_24670
> <Pooled Message Processor 2> tid=0x64] XXX
> ParallelQueueRemovalMessage.process bucket is not initialized and does not
> contain key destroying from tempQueue
> name=/__PR/_B__sender__bridgeds__1__to__bridgeds__2__PARALLEL__GATEWAY__SENDER__QUEUE_7;
> key=31555
> [warning 2016/10/19 22:55:30.048 UTC bridgegemfire_1_1_ip-10-32-109-74_24670
> <Pooled Message Processor 2> tid=0x64] XXX
> ParallelQueueRemovalMessage.destroyFromTempQueue destroyed
> qPR=sender_bridgeds_1_to_bridgeds_2_PARALLEL_GATEWAY_SENDER_QUEUE;
> bucketId=7; keyToRemove=31555
> {noformat}
> The bucket region contains the key right before it is available for use:
> {noformat}
> [warning 2016/10/19 22:55:30.239 UTC bridgegemfire_1_1_ip-10-32-109-74_24670
> <ResourceManagerRecoveryThread 0> tid=0x50] XXX
> BucketRegionQueue.cleanUpDestroyedTokensAndMarkGIIComplete
> _B__sender__bridgeds__1__to__bridgeds__2__PARALLEL__GATEWAY__SENDER__QUEUE_7
> keySet4=[33079, 31807, 32215, 33043, 32815, 31555, 32323, 32695, 32455,
> 32539, 32575, 32959, 32287, 32923, 32359, 32995, 33031, 32623, 33067, 32647,
> 32395, 32479, 33007, 32779, 32551, 32515, 32587, 32251, 32767, 32899, 32383,
> 32635, 32791, 32731, 32311, 32827, 32275, 32599, 32851, 32803, 32503, 32527,
> 32347, 32911, 32371, 32335]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
