[jira] [Commented] (GEODE-3055) data mismatch caused by rebalance. waitUntilFlashed return false
[
https://issues.apache.org/jira/browse/GEODE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135831#comment-16135831
]
ASF subversion and git services commented on GEODE-3055:
Commit d809076d01c28b9b819ab32d6af172004b3f8740 in geode's branch
refs/heads/feature/GEODE-3416 from zhouxh
[ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=d809076 ]
GEODE-3055: Should use a conservative fix to only catch the PartitionOfflineEx
to remove the leader region bucket.
Previous fix to catch all RuntimeException is too aggressive.
This closes #723
> data mismatch caused by rebalance. waitUntilFlashed return false
>
>
> Key: GEODE-3055
> URL: https://issues.apache.org/jira/browse/GEODE-3055
> Project: Geode
> Issue Type: Bug
>Reporter: xiaojian zhou
>Assignee: xiaojian zhou
> Labels: lucene
> Fix For: 1.3.0
>
>
> /export/buglogs_bvt/xzhou/lucene/concParRegHAPersist-0601-171739
> lucene/concParRegHAPersist.conf
> A=accessor
> B=dataStore
> accessorHosts=1
> accessorThreadsPerVM=5
> accessorVMsPerHost=1
> dataStoreHosts=6
> dataStoreThreadsPerVM=5
> dataStoreVMsPerHost=1
> numVMsToStop=2
> redundantCopies=0
> no local.conf
> In dataStoregemfire5_7483/system.log, thread tid=0xdf, putAll Object_11066
> 17:22:27.135 tid=0xdf] generated tag {v1; rv13 shadowKey=2939
> 17:22:27.136 _partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1 bucket : null //
> brq is not ready yet
> is enqueued to the tempQueue
> 17:22:27.272 tid=0xdf] generated tag {v3; rv15 shadowKey=3278
> 17:22:33.111 Subregion created:
> /_PR/_BAsyncEventQueueindex#partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1
> vm_3_dataStore3_r02-s28_28143.log:
> 17:22:33.120 Put successfully in the queue shadowKey= 2939
> 17:22:33.156 tid=0x7fe started query
> 17:22:33.176 Peeked shadowKey= 2939
> So the root cause is: the event is still in tempQueue before it's processed,
> the query happened. WaitUntilFlush should wait until tempQueue is also
> flushed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (GEODE-3055) data mismatch caused by rebalance. waitUntilFlashed return false
[
https://issues.apache.org/jira/browse/GEODE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133763#comment-16133763
]
ASF subversion and git services commented on GEODE-3055:
Commit d809076d01c28b9b819ab32d6af172004b3f8740 in geode's branch
refs/heads/develop from zhouxh
[ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=d809076 ]
GEODE-3055: Should use a conservative fix to only catch the PartitionOfflineEx
to remove the leader region bucket.
Previous fix to catch all RuntimeException is too aggressive.
This closes #723
> data mismatch caused by rebalance. waitUntilFlashed return false
>
>
> Key: GEODE-3055
> URL: https://issues.apache.org/jira/browse/GEODE-3055
> Project: Geode
> Issue Type: Bug
>Reporter: xiaojian zhou
>Assignee: xiaojian zhou
> Labels: lucene
> Fix For: 1.3.0
>
>
> /export/buglogs_bvt/xzhou/lucene/concParRegHAPersist-0601-171739
> lucene/concParRegHAPersist.conf
> A=accessor
> B=dataStore
> accessorHosts=1
> accessorThreadsPerVM=5
> accessorVMsPerHost=1
> dataStoreHosts=6
> dataStoreThreadsPerVM=5
> dataStoreVMsPerHost=1
> numVMsToStop=2
> redundantCopies=0
> no local.conf
> In dataStoregemfire5_7483/system.log, thread tid=0xdf, putAll Object_11066
> 17:22:27.135 tid=0xdf] generated tag {v1; rv13 shadowKey=2939
> 17:22:27.136 _partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1 bucket : null //
> brq is not ready yet
> is enqueued to the tempQueue
> 17:22:27.272 tid=0xdf] generated tag {v3; rv15 shadowKey=3278
> 17:22:33.111 Subregion created:
> /_PR/_BAsyncEventQueueindex#partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1
> vm_3_dataStore3_r02-s28_28143.log:
> 17:22:33.120 Put successfully in the queue shadowKey= 2939
> 17:22:33.156 tid=0x7fe started query
> 17:22:33.176 Peeked shadowKey= 2939
> So the root cause is: the event is still in tempQueue before it's processed,
> the query happened. WaitUntilFlush should wait until tempQueue is also
> flushed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (GEODE-3055) data mismatch caused by rebalance. waitUntilFlashed return false
[
https://issues.apache.org/jira/browse/GEODE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133758#comment-16133758
]
ASF subversion and git services commented on GEODE-3055:
Commit 360c3d536fb12106645d9942546324fc446546fe in geode's branch
refs/heads/feature/GEM-1601 from zhouxh
[ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=360c3d5 ]
GEODE-3055: Should use a conservative fix to only catch the PartitionOfflineEx
to remove the leader region bucket.
Previous fix to catch all RuntimeException is too aggressive.
> data mismatch caused by rebalance. waitUntilFlashed return false
>
>
> Key: GEODE-3055
> URL: https://issues.apache.org/jira/browse/GEODE-3055
> Project: Geode
> Issue Type: Bug
>Reporter: xiaojian zhou
>Assignee: xiaojian zhou
> Labels: lucene
> Fix For: 1.3.0
>
>
> /export/buglogs_bvt/xzhou/lucene/concParRegHAPersist-0601-171739
> lucene/concParRegHAPersist.conf
> A=accessor
> B=dataStore
> accessorHosts=1
> accessorThreadsPerVM=5
> accessorVMsPerHost=1
> dataStoreHosts=6
> dataStoreThreadsPerVM=5
> dataStoreVMsPerHost=1
> numVMsToStop=2
> redundantCopies=0
> no local.conf
> In dataStoregemfire5_7483/system.log, thread tid=0xdf, putAll Object_11066
> 17:22:27.135 tid=0xdf] generated tag {v1; rv13 shadowKey=2939
> 17:22:27.136 _partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1 bucket : null //
> brq is not ready yet
> is enqueued to the tempQueue
> 17:22:27.272 tid=0xdf] generated tag {v3; rv15 shadowKey=3278
> 17:22:33.111 Subregion created:
> /_PR/_BAsyncEventQueueindex#partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1
> vm_3_dataStore3_r02-s28_28143.log:
> 17:22:33.120 Put successfully in the queue shadowKey= 2939
> 17:22:33.156 tid=0x7fe started query
> 17:22:33.176 Peeked shadowKey= 2939
> So the root cause is: the event is still in tempQueue before it's processed,
> the query happened. WaitUntilFlush should wait until tempQueue is also
> flushed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (GEODE-3055) data mismatch caused by rebalance. waitUntilFlashed return false
[
https://issues.apache.org/jira/browse/GEODE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133671#comment-16133671
]
ASF subversion and git services commented on GEODE-3055:
Commit 291dac019f534af41470f0ee709c47477395d6fb in geode's branch
refs/heads/feature/GEM-1601 from zhouxh
[ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=291dac0 ]
GEODE-3055: Should use a conservative fix to only catch the PartitionOfflineEx
to remove the leader region bucket.
Previous fix to catch all RuntimeException is too aggressive.
> data mismatch caused by rebalance. waitUntilFlashed return false
>
>
> Key: GEODE-3055
> URL: https://issues.apache.org/jira/browse/GEODE-3055
> Project: Geode
> Issue Type: Bug
>Reporter: xiaojian zhou
>Assignee: xiaojian zhou
> Labels: lucene
> Fix For: 1.3.0
>
>
> /export/buglogs_bvt/xzhou/lucene/concParRegHAPersist-0601-171739
> lucene/concParRegHAPersist.conf
> A=accessor
> B=dataStore
> accessorHosts=1
> accessorThreadsPerVM=5
> accessorVMsPerHost=1
> dataStoreHosts=6
> dataStoreThreadsPerVM=5
> dataStoreVMsPerHost=1
> numVMsToStop=2
> redundantCopies=0
> no local.conf
> In dataStoregemfire5_7483/system.log, thread tid=0xdf, putAll Object_11066
> 17:22:27.135 tid=0xdf] generated tag {v1; rv13 shadowKey=2939
> 17:22:27.136 _partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1 bucket : null //
> brq is not ready yet
> is enqueued to the tempQueue
> 17:22:27.272 tid=0xdf] generated tag {v3; rv15 shadowKey=3278
> 17:22:33.111 Subregion created:
> /_PR/_BAsyncEventQueueindex#partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1
> vm_3_dataStore3_r02-s28_28143.log:
> 17:22:33.120 Put successfully in the queue shadowKey= 2939
> 17:22:33.156 tid=0x7fe started query
> 17:22:33.176 Peeked shadowKey= 2939
> So the root cause is: the event is still in tempQueue before it's processed,
> the query happened. WaitUntilFlush should wait until tempQueue is also
> flushed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (GEODE-3055) data mismatch caused by rebalance. waitUntilFlashed return false
[
https://issues.apache.org/jira/browse/GEODE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129607#comment-16129607
]
ASF subversion and git services commented on GEODE-3055:
Commit 2f61dd600eec36f2588ec82a7cfdf1b6e27057f9 in geode's branch
refs/heads/develop from zhouxh
[ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=2f61dd6 ]
GEODE-3055: change the error message's log level from info to debug.
> data mismatch caused by rebalance. waitUntilFlashed return false
>
>
> Key: GEODE-3055
> URL: https://issues.apache.org/jira/browse/GEODE-3055
> Project: Geode
> Issue Type: Bug
>Reporter: xiaojian zhou
>Assignee: xiaojian zhou
> Labels: lucene
> Fix For: 1.3.0
>
>
> /export/buglogs_bvt/xzhou/lucene/concParRegHAPersist-0601-171739
> lucene/concParRegHAPersist.conf
> A=accessor
> B=dataStore
> accessorHosts=1
> accessorThreadsPerVM=5
> accessorVMsPerHost=1
> dataStoreHosts=6
> dataStoreThreadsPerVM=5
> dataStoreVMsPerHost=1
> numVMsToStop=2
> redundantCopies=0
> no local.conf
> In dataStoregemfire5_7483/system.log, thread tid=0xdf, putAll Object_11066
> 17:22:27.135 tid=0xdf] generated tag {v1; rv13 shadowKey=2939
> 17:22:27.136 _partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1 bucket : null //
> brq is not ready yet
> is enqueued to the tempQueue
> 17:22:27.272 tid=0xdf] generated tag {v3; rv15 shadowKey=3278
> 17:22:33.111 Subregion created:
> /_PR/_BAsyncEventQueueindex#partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1
> vm_3_dataStore3_r02-s28_28143.log:
> 17:22:33.120 Put successfully in the queue shadowKey= 2939
> 17:22:33.156 tid=0x7fe started query
> 17:22:33.176 Peeked shadowKey= 2939
> So the root cause is: the event is still in tempQueue before it's processed,
> the query happened. WaitUntilFlush should wait until tempQueue is also
> flushed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (GEODE-3055) data mismatch caused by rebalance. waitUntilFlashed return false
[
https://issues.apache.org/jira/browse/GEODE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111398#comment-16111398
]
ASF subversion and git services commented on GEODE-3055:
Commit bb847c767f1507eed8c26dee02b377c65d43a0f7 in geode's branch
refs/heads/develop from zhouxh
[ https://git-wip-us.apache.org/repos/asf?p=geode.git;h=bb847c7 ]
GEODE-3055: The old primary's the shadow bucket is not
initialized when rebalance remove it. Thus the new primary candidate can
never initialize from it. The fix is to wait until new primary exists before
remove the old primary's bucket in rebalance.
When the colocated child bucket failed to initialize, remove the leader
bucket since all these buckets should be created atomically.
This closes #570
> data mismatch caused by rebalance. waitUntilFlashed return false
>
>
> Key: GEODE-3055
> URL: https://issues.apache.org/jira/browse/GEODE-3055
> Project: Geode
> Issue Type: Bug
>Reporter: xiaojian zhou
>Assignee: xiaojian zhou
> Labels: lucene
>
> /export/buglogs_bvt/xzhou/lucene/concParRegHAPersist-0601-171739
> lucene/concParRegHAPersist.conf
> A=accessor
> B=dataStore
> accessorHosts=1
> accessorThreadsPerVM=5
> accessorVMsPerHost=1
> dataStoreHosts=6
> dataStoreThreadsPerVM=5
> dataStoreVMsPerHost=1
> numVMsToStop=2
> redundantCopies=0
> no local.conf
> In dataStoregemfire5_7483/system.log, thread tid=0xdf, putAll Object_11066
> 17:22:27.135 tid=0xdf] generated tag {v1; rv13 shadowKey=2939
> 17:22:27.136 _partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1 bucket : null //
> brq is not ready yet
> is enqueued to the tempQueue
> 17:22:27.272 tid=0xdf] generated tag {v3; rv15 shadowKey=3278
> 17:22:33.111 Subregion created:
> /_PR/_BAsyncEventQueueindex#partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1
> vm_3_dataStore3_r02-s28_28143.log:
> 17:22:33.120 Put successfully in the queue shadowKey= 2939
> 17:22:33.156 tid=0x7fe started query
> 17:22:33.176 Peeked shadowKey= 2939
> So the root cause is: the event is still in tempQueue before it's processed,
> the query happened. WaitUntilFlush should wait until tempQueue is also
> flushed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (GEODE-3055) data mismatch caused by rebalance. waitUntilFlashed return false
[
https://issues.apache.org/jira/browse/GEODE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111343#comment-16111343
]
ASF GitHub Bot commented on GEODE-3055:
---
Github user gesterzhou commented on a diff in the pull request:
https://github.com/apache/geode/pull/570#discussion_r130938862
--- Diff:
geode-core/src/main/java/org/apache/geode/internal/cache/PartitionedRegionDataStore.java
---
@@ -494,6 +494,18 @@ CreateBucketResult grabFreeBucket(final int
possiblyFreeBucketId, final Distribu
return result;
+} catch (RuntimeException validationException) {
+ // GEODE-3055
+ PartitionedRegion leader =
ColocationHelper.getLeaderRegion(this.partitionedRegion);
+ boolean isLeader = leader.equals(this.partitionedRegion);
--- End diff --
Remove the leader will remove all its children.
> data mismatch caused by rebalance. waitUntilFlashed return false
>
>
> Key: GEODE-3055
> URL: https://issues.apache.org/jira/browse/GEODE-3055
> Project: Geode
> Issue Type: Bug
>Reporter: xiaojian zhou
>Assignee: xiaojian zhou
> Labels: lucene
>
> /export/buglogs_bvt/xzhou/lucene/concParRegHAPersist-0601-171739
> lucene/concParRegHAPersist.conf
> A=accessor
> B=dataStore
> accessorHosts=1
> accessorThreadsPerVM=5
> accessorVMsPerHost=1
> dataStoreHosts=6
> dataStoreThreadsPerVM=5
> dataStoreVMsPerHost=1
> numVMsToStop=2
> redundantCopies=0
> no local.conf
> In dataStoregemfire5_7483/system.log, thread tid=0xdf, putAll Object_11066
> 17:22:27.135 tid=0xdf] generated tag {v1; rv13 shadowKey=2939
> 17:22:27.136 _partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1 bucket : null //
> brq is not ready yet
> is enqueued to the tempQueue
> 17:22:27.272 tid=0xdf] generated tag {v3; rv15 shadowKey=3278
> 17:22:33.111 Subregion created:
> /_PR/_BAsyncEventQueueindex#partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1
> vm_3_dataStore3_r02-s28_28143.log:
> 17:22:33.120 Put successfully in the queue shadowKey= 2939
> 17:22:33.156 tid=0x7fe started query
> 17:22:33.176 Peeked shadowKey= 2939
> So the root cause is: the event is still in tempQueue before it's processed,
> the query happened. WaitUntilFlush should wait until tempQueue is also
> flushed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (GEODE-3055) data mismatch caused by rebalance. waitUntilFlashed return false
[
https://issues.apache.org/jira/browse/GEODE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111335#comment-16111335
]
ASF GitHub Bot commented on GEODE-3055:
---
Github user boglesby commented on a diff in the pull request:
https://github.com/apache/geode/pull/570#discussion_r130937683
--- Diff:
geode-core/src/main/java/org/apache/geode/internal/cache/PartitionedRegionDataStore.java
---
@@ -494,6 +494,18 @@ CreateBucketResult grabFreeBucket(final int
possiblyFreeBucketId, final Distribu
return result;
+} catch (RuntimeException validationException) {
+ // GEODE-3055
+ PartitionedRegion leader =
ColocationHelper.getLeaderRegion(this.partitionedRegion);
+ boolean isLeader = leader.equals(this.partitionedRegion);
--- End diff --
What if there is more than one colocated child region and one of the child
buckets is successfully created, but the other is not. This code will remove
the parent bucket. Will the one that was successfully created also get removed?
> data mismatch caused by rebalance. waitUntilFlashed return false
>
>
> Key: GEODE-3055
> URL: https://issues.apache.org/jira/browse/GEODE-3055
> Project: Geode
> Issue Type: Bug
>Reporter: xiaojian zhou
>Assignee: xiaojian zhou
> Labels: lucene
>
> /export/buglogs_bvt/xzhou/lucene/concParRegHAPersist-0601-171739
> lucene/concParRegHAPersist.conf
> A=accessor
> B=dataStore
> accessorHosts=1
> accessorThreadsPerVM=5
> accessorVMsPerHost=1
> dataStoreHosts=6
> dataStoreThreadsPerVM=5
> dataStoreVMsPerHost=1
> numVMsToStop=2
> redundantCopies=0
> no local.conf
> In dataStoregemfire5_7483/system.log, thread tid=0xdf, putAll Object_11066
> 17:22:27.135 tid=0xdf] generated tag {v1; rv13 shadowKey=2939
> 17:22:27.136 _partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1 bucket : null //
> brq is not ready yet
> is enqueued to the tempQueue
> 17:22:27.272 tid=0xdf] generated tag {v3; rv15 shadowKey=3278
> 17:22:33.111 Subregion created:
> /_PR/_BAsyncEventQueueindex#partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1
> vm_3_dataStore3_r02-s28_28143.log:
> 17:22:33.120 Put successfully in the queue shadowKey= 2939
> 17:22:33.156 tid=0x7fe started query
> 17:22:33.176 Peeked shadowKey= 2939
> So the root cause is: the event is still in tempQueue before it's processed,
> the query happened. WaitUntilFlush should wait until tempQueue is also
> flushed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (GEODE-3055) data mismatch caused by rebalance. waitUntilFlashed return false
[
https://issues.apache.org/jira/browse/GEODE-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109783#comment-16109783
]
ASF GitHub Bot commented on GEODE-3055:
---
Github user gesterzhou commented on a diff in the pull request:
https://github.com/apache/geode/pull/570#discussion_r130729615
--- Diff:
geode-core/src/main/java/org/apache/geode/internal/cache/PartitionedRegionDataStore.java
---
@@ -1472,6 +1472,19 @@ public boolean removeBucket(int bucketId, boolean
forceRemovePrimary) {
}
BucketAdvisor bucketAdvisor = bucketRegion.getBucketAdvisor();
+ InternalDistributedMember primary = bucketAdvisor.getPrimary();
+ InternalDistributedMember myId =
+
this.partitionedRegion.getDistributionManager().getDistributionManagerId();
+ if (primary == null || myId.equals(primary)) {
--- End diff --
The forceRemovePrimary WAS useless and it can be removed because it always
use "false".
But when I added the logic to remove the leader region bucket (when the
shadow bucket failed to initialize), then I have to call the removeBucket(xxx,
true) by myself.
The shadow bucket was removed first by the exception handling. But I added
the logic to remove leader bucket, I have to skip a few "return false" exit
points, because at this time the leader bucket is not logically ready and not
qualified to be removed unless I force to remove it.
So I make use of the forceRemovePrimary parameter. Maybe I should change it
to better name, such as forceToRemove.
> data mismatch caused by rebalance. waitUntilFlashed return false
>
>
> Key: GEODE-3055
> URL: https://issues.apache.org/jira/browse/GEODE-3055
> Project: Geode
> Issue Type: Bug
>Reporter: xiaojian zhou
>Assignee: xiaojian zhou
> Labels: lucene
>
> /export/buglogs_bvt/xzhou/lucene/concParRegHAPersist-0601-171739
> lucene/concParRegHAPersist.conf
> A=accessor
> B=dataStore
> accessorHosts=1
> accessorThreadsPerVM=5
> accessorVMsPerHost=1
> dataStoreHosts=6
> dataStoreThreadsPerVM=5
> dataStoreVMsPerHost=1
> numVMsToStop=2
> redundantCopies=0
> no local.conf
> In dataStoregemfire5_7483/system.log, thread tid=0xdf, putAll Object_11066
> 17:22:27.135 tid=0xdf] generated tag {v1; rv13 shadowKey=2939
> 17:22:27.136 _partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1 bucket : null //
> brq is not ready yet
> is enqueued to the tempQueue
> 17:22:27.272 tid=0xdf] generated tag {v3; rv15 shadowKey=3278
> 17:22:33.111 Subregion created:
> /_PR/_BAsyncEventQueueindex#partitionedRegionPARALLELGATEWAYSENDER_QUEUE_1
> vm_3_dataStore3_r02-s28_28143.log:
> 17:22:33.120 Put successfully in the queue shadowKey= 2939
> 17:22:33.156 tid=0x7fe started query
> 17:22:33.176 Peeked shadowKey= 2939
> So the root cause is: the event is still in tempQueue before it's processed,
> the query happened. WaitUntilFlush should wait until tempQueue is also
> flushed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
