[ https://issues.apache.org/jira/browse/GEODE-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612392#comment-15612392 ]
ASF subversion and git services commented on GEODE-2027: -------------------------------------------------------- Commit b10a171e11990a566ed42560a903668908131390 in incubator-geode's branch refs/heads/develop from [~barry.oglesby] [ https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=b10a171 ] GEODE-2027: ParallelQueueRemovalMessage processing removes events from the region and temp queue > Gateway events being processed by the ParallelQueueRemovalMessage in the > secondary at the time of GII may not be removed from the queue successfully > ---------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: GEODE-2027 > URL: https://issues.apache.org/jira/browse/GEODE-2027 > Project: Geode > Issue Type: Bug > Components: wan > Reporter: Barry Oglesby > > This is the same as Trac ticket 49196. > I see two cases that cause this issue (so far). In both cases the member is > starting and GIIing the bucket region, and the entry in question is included > in the GII response. > In case one: > - the bucket region is not found in the {{ParallelGatewaySenderQueue.put}} > path, so the event is added to the temp queue > - the bucket region is found in the {{ParallelQueueRemovalMessage.process}} > path and contains the key so it is removed from it > - the event is added back into the bucket region from the temp queue > The event is added to the temp queue: > {noformat} > [warning 2016/10/20 01:25:07.715 UTC bridgegemfire_1_1_ip-10-32-109-74_25707 > <P2P message reader for > ip-10-32-109-74(bridgegemfire_1_2_ip-10-32-109-74_25713:25713)<ec><v1>:1026 > unshared ordered uid=75 dom #1 port=57564> tid=0x82] XXX > ParallelGatewaySenderQueue.put adding to tempQueue brq=null; key=33283; > value=SenderEventImpl[id=EventID[threadID=0x10007|5;sequenceID=177;bucketId=7];action=2;operation=DESTROY;region=/testRegion1;key=Object_2934;value=;valueIsObject=1;numberOfParts=8;callbackArgument=GatewaySenderEventCallbackArgument > > [originalCallbackArg=null;originatingSenderId=1;recipientGatewayReceivers=[2]];possibleDuplicate=false;creationTime=1476926707714;shadowKey= > 33283;timeStamp=1476926707709;acked=false;dispatched=false] > {noformat} > The event exists in the bucket region and is removed: > {noformat} > [warning 2016/10/20 01:25:10.265 UTC bridgegemfire_1_1_ip-10-32-109-74_25707 > <Pooled Message Processor 1> tid=0x63] XXX > ParallelQueueRemovalMessage.process bucket is not initialized but contains > key destroying from bucket > name=/__PR/_B__sender__bridgeds__1__to__bridgeds__2__PARALLEL__GATEWAY__SENDER__QUEUE_7; > key=33283 > {noformat} > The event is loaded from the temp queue into the bucket region: > {noformat} > [warning 2016/10/20 01:25:10.432 UTC bridgegemfire_1_1_ip-10-32-109-74_25707 > <ResourceManagerRecoveryThread 0> tid=0x4f] XXX > AbstractBucketRegionQueue.loadEventsFromTempQueue > _B__sender__bridgeds__1__to__bridgeds__2__PARALLEL__GATEWAY__SENDER__QUEUE_7 > adding to queue key=33283 -> > SenderEventImpl[id=EventID[threadID=0x10007|5;sequenceID=177;bucketId=7];action=2;operation=DESTROY;region=/testRegion1;key=Object_2934;value=;valueIsObject=1;numberOfParts=8;callbackArgument=GatewaySenderEventCallbackArgument > > [originalCallbackArg=null;originatingSenderId=1;recipientGatewayReceivers=[2]];possibleDuplicate=true;creationTime=1476926707714;shadowKey= > 33283;timeStamp=1476926707709;acked=false;dispatched=false] > {noformat} > The bucket region contains the key right before it is available for use: > {noformat} > [warning 2016/10/20 01:25:10.790 UTC bridgegemfire_1_1_ip-10-32-109-74_25707 > <ResourceManagerRecoveryThread 0> tid=0x4f] XXX > BucketRegionQueue.cleanUpDestroyedTokensAndMarkGIIComplete > _B__sender__bridgeds__1__to__bridgeds__2__PARALLEL__GATEWAY__SENDER__QUEUE_7 > keySet4=[33283, 33079, 34591, 35071, 35119, 34879, 34447, 34891, 34843, > 35083, 33295, 34387, 34339, 32743, 35131, 35095, 35059, 34603, 34471, 34567, > 34375, 32983, 32875, 34399, 34867, 34831, 34759, 34675, 35107, 35047, 34723, > 34495, 34507, 34459, 34699, 34735] > {noformat} > In case two: > - the bucket region is not found in the {{ParallelGatewaySenderQueue.put}} > path, so the event is added to the temp queue > - the bucket region is found in the {{ParallelQueueRemovalMessage.process}} > path, but doesn't contain the key so it is removed from the temp queue > - the event is removed from the temp queue but is also in the bucket region > The event is added to the temp queue: > {noformat} > [warning 2016/10/19 22:55:27.743 UTC bridgegemfire_1_1_ip-10-32-109-74_24670 > <P2P message reader for > ip-10-32-109-74(bridgegemfire_1_3_ip-10-32-109-74_24716:24716)<ec><v1>:1028 > unshared ordered uid=62 dom #1 port=56962> tid=0x68] XXX > ParallelGatewaySenderQueue.put adding to tempQueue brq=null; key=31555; > value=SenderEventImpl[id=EventID[threadID=0x10007|7;sequenceID=108;bucketId=7];action=0;operation=CREATE;region=/testRegion1;key=Object_10236;value=util.ValueHolder > [myVersion=util.ValueHolder, myValue=<java.lang.Long, value 10236>, > extraObject=<[B Size 70, componentType byte>, > modVal=null];valueIsObject=1;numberOfParts=9;callbackArgument=GatewaySenderEventCallbackArgument > > [originalCallbackArg=July;originatingSenderId=1;recipientGatewayReceivers=[2]];possibleDuplicate=false;creationTime=1476917727742;shadowKey= > 31555;timeStamp=1476917727723;acked=false;dispatched=false] > {noformat} > The event doesn't exist in the bucket region and is removed from the temp > queue: > {noformat} > [warning 2016/10/19 22:55:30.047 UTC bridgegemfire_1_1_ip-10-32-109-74_24670 > <Pooled Message Processor 2> tid=0x64] XXX > ParallelQueueRemovalMessage.process bucket is not initialized and does not > contain key destroying from tempQueue > name=/__PR/_B__sender__bridgeds__1__to__bridgeds__2__PARALLEL__GATEWAY__SENDER__QUEUE_7; > key=31555 > [warning 2016/10/19 22:55:30.048 UTC bridgegemfire_1_1_ip-10-32-109-74_24670 > <Pooled Message Processor 2> tid=0x64] XXX > ParallelQueueRemovalMessage.destroyFromTempQueue destroyed > qPR=sender_bridgeds_1_to_bridgeds_2_PARALLEL_GATEWAY_SENDER_QUEUE; > bucketId=7; keyToRemove=31555 > {noformat} > The bucket region contains the key right before it is available for use: > {noformat} > [warning 2016/10/19 22:55:30.239 UTC bridgegemfire_1_1_ip-10-32-109-74_24670 > <ResourceManagerRecoveryThread 0> tid=0x50] XXX > BucketRegionQueue.cleanUpDestroyedTokensAndMarkGIIComplete > _B__sender__bridgeds__1__to__bridgeds__2__PARALLEL__GATEWAY__SENDER__QUEUE_7 > keySet4=[33079, 31807, 32215, 33043, 32815, 31555, 32323, 32695, 32455, > 32539, 32575, 32959, 32287, 32923, 32359, 32995, 33031, 32623, 33067, 32647, > 32395, 32479, 33007, 32779, 32551, 32515, 32587, 32251, 32767, 32899, 32383, > 32635, 32791, 32731, 32311, 32827, 32275, 32599, 32851, 32803, 32503, 32527, > 32347, 32911, 32371, 32335] > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)