[
https://issues.apache.org/jira/browse/GEODE-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353895#comment-15353895
]
ASF GitHub Bot commented on GEODE-1558:
---------------------------------------
GitHub user aparnard opened a pull request:
https://github.com/apache/incubator-geode/pull/176
GEODE-1558: Fixing failures due to rebalancing buckets during lucene …
…index updates
There are two failures we're fixing here:
1) Calling repository.create for create events was leaving duplicate events
in the
index because when a bucket fails over, the event is dispatched again on
the new primary.
Using the isPossibleDuplicate flag did not work because it was not
consistently set to
true for duplicate events. Changed the code to call repository.update even
for create events
2) The async event queue was repeatedly dispatching the same events event
after
a bucket moved to another node. We changed the async event queue code to
filter out
events for buckets that are no longer present on this dispatching member.
Cleaning up the rebalancing test and adding new tests to make these
scenarios
more reproducable.
Signed-off-by: Aparna Dharmakkan <[email protected]>
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aparnard/incubator-geode feature/GEODE-1558
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-geode/pull/176.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #176
----
commit d659a4ae97602e885638dd4081250aee02708a15
Author: Dan Smith <[email protected]>
Date: 2016-06-28T22:33:51Z
GEODE-1558: Fixing failures due to rebalancing buckets during lucene index
updates
There are two failures we're fixing here:
1) Calling repository.create for create events was leaving duplicate events
in the
index because when a bucket fails over, the event is dispatched again on
the new primary.
Using the isPossibleDuplicate flag did not work because it was not
consistently set to
true for duplicate events. Changed the code to call repository.update even
for create events
2) The async event queue was repeatedly dispatching the same events event
after
a bucket moved to another node. We changed the async event queue code to
filter out
events for buckets that are no longer present on this dispatching member.
Cleaning up the rebalancing test and adding new tests to make these
scenarios
more reproducable.
Signed-off-by: Aparna Dharmakkan <[email protected]>
----
> LuceneQueriesPeerPRDUnitTest.returnCorrectResultsAfterRebalance failed on
> flushing the queue
> --------------------------------------------------------------------------------------------
>
> Key: GEODE-1558
> URL: https://issues.apache.org/jira/browse/GEODE-1558
> Project: Geode
> Issue Type: Bug
> Components: lucene
> Reporter: Aparna Dharmakkan
>
> The error was:
> {noformat}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> com.gemstone.gemfire.cache.lucene.LuceneQueriesPRBase.returnCorrectResultsAfterRebalance(LuceneQueriesPRBase.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:112)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
> at
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
> at
> org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> and the log output:
> {noformat}
> [vm_0]49.448: [GC (Allocation Failure) [PSYoungGen: 150556K->11769K(162816K)]
> 165672K->37374K(230400K), 0.0693682 secs] [Times: user=0.06 sys=0.01,
> real=0.07 secs]
> [vm_3][warn 2016/06/16 16:08:16.915 PDT <Event Processor for
> GatewaySender_AsyncEventQueue_index#_region_0> tid=0x48]
> com.gemstone.gemfire.internal.cache.wan.GatewaySenderEventCallbackDispatcher@1215f139:
> Exception during processing batch 0
> [vm_3]com.gemstone.gemfire.internal.cache.wan.GatewaySenderException:
> com.gemstone.gemfire.internal.cache.wan.GatewaySenderEventCallbackDispatcher@1215f139:
> Exception during processing batch 0, caused by
> com.gemstone.gemfire.internal.cache.PrimaryBucketException: Bucket
> _B__index#__region.files_0 is not primary. Current primary holder is
> 192.168.1.17(12468)<ec><v51>:1027
> [vm_3] at
> com.gemstone.gemfire.internal.cache.wan.GatewaySenderEventCallbackDispatcher.dispatchBatch(GatewaySenderEventCallbackDispatcher.java:177)
> [vm_3] at
> com.gemstone.gemfire.internal.cache.wan.GatewaySenderEventCallbackDispatcher.dispatchBatch(GatewaySenderEventCallbackDispatcher.java:87)
> [vm_3] at
> com.gemstone.gemfire.internal.cache.wan.AbstractGatewaySenderEventProcessor.processQueue(AbstractGatewaySenderEventProcessor.java:613)
> [vm_3] at
> com.gemstone.gemfire.internal.cache.wan.AbstractGatewaySenderEventProcessor.run(AbstractGatewaySenderEventProcessor.java:1058)
> [vm_3]Caused by: com.gemstone.gemfire.internal.cache.PrimaryBucketException:
> Bucket _B__index#__region.files_0 is not primary. Current primary holder is
> 192.168.1.17(12468)<ec><v51>:1027
> [vm_3] at
> com.gemstone.gemfire.internal.cache.BucketRegion.checkForPrimary(BucketRegion.java:670)
> [vm_3] at
> com.gemstone.gemfire.internal.cache.BucketRegion.doLockForPrimary(BucketRegion.java:721)
> [vm_3] at
> com.gemstone.gemfire.internal.cache.BucketRegion.beginLocalWrite(BucketRegion.java:697)
> [vm_3] at
> com.gemstone.gemfire.internal.cache.BucketRegion.virtualPut(BucketRegion.java:476)
> [vm_3] at
> com.gemstone.gemfire.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:132)
> [vm_3] at
> com.gemstone.gemfire.internal.cache.LocalRegion.basicPut(LocalRegion.java:5241)
> [vm_3] at
> com.gemstone.gemfire.internal.cache.LocalRegion.putIfAbsent(LocalRegion.java:12022)
> [vm_3] at
> com.gemstone.gemfire.internal.cache.LocalRegion.putIfAbsent(LocalRegion.java:12046)
> [vm_3] at
> com.gemstone.gemfire.cache.lucene.internal.filesystem.FileSystem.createFile(FileSystem.java:67)
> [vm_3] at
> com.gemstone.gemfire.cache.lucene.internal.directory.RegionDirectory.createOutput(RegionDirectory.java:84)
> [vm_3] at
> org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:44)
> [vm_3] at
> org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
> [vm_3] at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:108)
> [vm_3] at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:128)
> [vm_3] at
> org.apache.lucene.codecs.lucene50.Lucene50StoredFieldsFormat.fieldsWriter(Lucene50StoredFieldsFormat.java:183)
> [vm_3] at
> org.apache.lucene.index.DefaultIndexingChain.initStoredFieldsWriter(DefaultIndexingChain.java:83)
> [vm_3] at
> org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:331)
> [vm_3] at
> org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:368)
> [vm_3] at
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
> [vm_3] at
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:449)
> [vm_3] at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1492)
> [vm_3] at
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1271)
> [vm_3] at
> com.gemstone.gemfire.cache.lucene.internal.repository.IndexRepositoryImpl.create(IndexRepositoryImpl.java:70)
> [vm_3] at
> com.gemstone.gemfire.cache.lucene.internal.LuceneEventListener.processEvents(LuceneEventListener.java:78)
> [vm_3] at
> com.gemstone.gemfire.internal.cache.wan.GatewaySenderEventCallbackDispatcher.dispatchBatch(GatewaySenderEventCallbackDispatcher.java:168)
> [vm_3] ... 3 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
