[ https://issues.apache.org/jira/browse/GEODE-774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
xiaojian zhou updated GEODE-774: -------------------------------- Description: {noformat} This is the 52550 issue we simulated and reproduced in house. CLIENT vm_4_thr_9_bridge_1_2_w2-2013-lin-04_6672 TASK[0] newWan.WANTest.HydraTask_doOps ERROR util.TestException: com.gemstone.gemfire.cache.CacheClosedException: For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by com.gemstone.gemfire.cache.DiskAccessException: For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by java.lang.AssertionError: Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null) at com.gemstone.gemfire.internal.cache.GemFireCacheImpl$Stopper.generateCancelledException(GemFireCacheImpl.java:1464) at com.gemstone.gemfire.CancelCriterion.checkCancelInProgress(CancelCriterion.java:91) at com.gemstone.gemfire.internal.cache.LocalRegion.checkRegionDestroyed(LocalRegion.java:8118) at com.gemstone.gemfire.internal.cache.LocalRegion.checkReadiness(LocalRegion.java:2994) at com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet$KeysSetIterator.next(PartitionedRegion.java:6544) at com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet.toArray(PartitionedRegion.java:6638) at com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet.toArray(PartitionedRegion.java:6631) at java.util.Collections$UnmodifiableCollection.toArray(Collections.java:1033) at java.util.ArrayList.addAll(ArrayList.java:577) at newWan.WANOperationsClient.getExistingKeys(WANOperationsClient.java:792) at newWan.WANOperationsClient.putAll(WANOperationsClient.java:463) at newWan.WANOperationsClient.doOps(WANOperationsClient.java:1419) at newWan.WANTest.doOps(WANTest.java:1650) at newWan.WANTest.HydraTask_doOps(WANTest.java:1639) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at hydra.MethExecutor.execute(MethExecutor.java:198) at hydra.MethExecutor.execute(MethExecutor.java:162) at hydra.TestTask.execute(TestTask.java:195) at hydra.RemoteTestModule$1.run(RemoteTestModule.java:216) Caused by: com.gemstone.gemfire.cache.DiskAccessException: For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by java.lang.AssertionError: Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null) at com.gemstone.gemfire.internal.cache.DiskStoreImpl$FlusherThread.run(DiskStoreImpl.java:1812) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.AssertionError: Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null) at com.gemstone.gemfire.internal.cache.Oplog.create(Oplog.java:3553) at com.gemstone.gemfire.internal.cache.PersistentOplogSet.create(PersistentOplogSet.java:189) at com.gemstone.gemfire.internal.cache.DiskStoreImpl.put(DiskStoreImpl.java:751) at com.gemstone.gemfire.internal.cache.DiskRegion.put(DiskRegion.java:381) at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeBytesToDisk(DiskEntry.java:874) at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeToDisk(DiskEntry.java:867) at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeToDisk(DiskEntry.java:857) at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.doAsyncFlush(DiskEntry.java:1742) at com.gemstone.gemfire.internal.cache.DiskStoreImpl$FlusherThread.run(DiskStoreImpl.java:1768) ... 1 more [CIRCULAR REFERENCE:com.gemstone.gemfire.cache.DiskAccessException: For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by java.lang.AssertionError: Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null)] To reproduce it, run following bt test: rollingupgrade/newWan/parallel/garminHCT.conf bridgeHostsPerSite=2 bridgeThreadsPerVM=2 bridgeVMsPerHost=4 clientMem=2g edgeHostsPerSite=2 edgeThreadsPerVM=5 edgeVMsPerHost=5 locatorHostsPerSite=2 locatorThreadsPerVM=1 locatorVMsPerHost=1 resultWaitSec=1200 serverMem=12g wanSites=2 with local.conf: newWan.WANOperationsClientPrms-randModValueForDebug = true; hydra.HostPrms-hostNames = fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\", 1, 4)" ncf fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\", 1, 4)" ncf fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\", 1, 4)" ncf ; hydra.HostPrms-resourceDirBaseMapFileName = /home/xzhou/bin/hostdirs.prop; hydra.RegionPrms-cacheListeners = none; {noformat} was: {noformat} This is the garmin issue we simulated and reproduced in house. CLIENT vm_4_thr_9_bridge_1_2_w2-2013-lin-04_6672 TASK[0] newWan.WANTest.HydraTask_doOps ERROR util.TestException: com.gemstone.gemfire.cache.CacheClosedException: For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by com.gemstone.gemfire.cache.DiskAccessException: For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by java.lang.AssertionError: Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null) at com.gemstone.gemfire.internal.cache.GemFireCacheImpl$Stopper.generateCancelledException(GemFireCacheImpl.java:1464) at com.gemstone.gemfire.CancelCriterion.checkCancelInProgress(CancelCriterion.java:91) at com.gemstone.gemfire.internal.cache.LocalRegion.checkRegionDestroyed(LocalRegion.java:8118) at com.gemstone.gemfire.internal.cache.LocalRegion.checkReadiness(LocalRegion.java:2994) at com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet$KeysSetIterator.next(PartitionedRegion.java:6544) at com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet.toArray(PartitionedRegion.java:6638) at com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet.toArray(PartitionedRegion.java:6631) at java.util.Collections$UnmodifiableCollection.toArray(Collections.java:1033) at java.util.ArrayList.addAll(ArrayList.java:577) at newWan.WANOperationsClient.getExistingKeys(WANOperationsClient.java:792) at newWan.WANOperationsClient.putAll(WANOperationsClient.java:463) at newWan.WANOperationsClient.doOps(WANOperationsClient.java:1419) at newWan.WANTest.doOps(WANTest.java:1650) at newWan.WANTest.HydraTask_doOps(WANTest.java:1639) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at hydra.MethExecutor.execute(MethExecutor.java:198) at hydra.MethExecutor.execute(MethExecutor.java:162) at hydra.TestTask.execute(TestTask.java:195) at hydra.RemoteTestModule$1.run(RemoteTestModule.java:216) Caused by: com.gemstone.gemfire.cache.DiskAccessException: For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by java.lang.AssertionError: Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null) at com.gemstone.gemfire.internal.cache.DiskStoreImpl$FlusherThread.run(DiskStoreImpl.java:1812) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.AssertionError: Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null) at com.gemstone.gemfire.internal.cache.Oplog.create(Oplog.java:3553) at com.gemstone.gemfire.internal.cache.PersistentOplogSet.create(PersistentOplogSet.java:189) at com.gemstone.gemfire.internal.cache.DiskStoreImpl.put(DiskStoreImpl.java:751) at com.gemstone.gemfire.internal.cache.DiskRegion.put(DiskRegion.java:381) at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeBytesToDisk(DiskEntry.java:874) at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeToDisk(DiskEntry.java:867) at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeToDisk(DiskEntry.java:857) at com.gemstone.gemfire.internal.cache.DiskEntry$Helper.doAsyncFlush(DiskEntry.java:1742) at com.gemstone.gemfire.internal.cache.DiskStoreImpl$FlusherThread.run(DiskStoreImpl.java:1768) ... 1 more [CIRCULAR REFERENCE:com.gemstone.gemfire.cache.DiskAccessException: For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by java.lang.AssertionError: Version stamp should have a member at this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; time=1452091668017};member=null)] To reproduce it, run following bt test: rollingupgrade/newWan/parallel/garminHCT.conf bridgeHostsPerSite=2 bridgeThreadsPerVM=2 bridgeVMsPerHost=4 clientMem=2g edgeHostsPerSite=2 edgeThreadsPerVM=5 edgeVMsPerHost=5 locatorHostsPerSite=2 locatorThreadsPerVM=1 locatorVMsPerHost=1 resultWaitSec=1200 serverMem=12g wanSites=2 with local.conf: newWan.WANOperationsClientPrms-randModValueForDebug = true; hydra.HostPrms-hostNames = fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\", 1, 4)" ncf fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\", 1, 4)" ncf fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\", 1, 4)" ncf ; hydra.HostPrms-resourceDirBaseMapFileName = /home/xzhou/bin/hostdirs.prop; hydra.RegionPrms-cacheListeners = none; {noformat} > WAN Cluster outgage due to DiskAccessException > ---------------------------------------------- > > Key: GEODE-774 > URL: https://issues.apache.org/jira/browse/GEODE-774 > Project: Geode > Issue Type: Bug > Reporter: xiaojian zhou > > {noformat} > This is the 52550 issue we simulated and reproduced in house. > CLIENT vm_4_thr_9_bridge_1_2_w2-2013-lin-04_6672 > TASK[0] newWan.WANTest.HydraTask_doOps > ERROR util.TestException: com.gemstone.gemfire.cache.CacheClosedException: > For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by > com.gemstone.gemfire.cache.DiskAccessException: For DiskStore: disk3: Fatal > error from asynchronous flusher thread, caused by java.lang.AssertionError: > Version stamp should have a member at this point for entry > VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a (key=Object_36851; > rawValue=VMCachedDeserializable@454500130; version={v0; rv0; ds=0; > time=1452091668017};member=null) > at > com.gemstone.gemfire.internal.cache.GemFireCacheImpl$Stopper.generateCancelledException(GemFireCacheImpl.java:1464) > at > com.gemstone.gemfire.CancelCriterion.checkCancelInProgress(CancelCriterion.java:91) > at > com.gemstone.gemfire.internal.cache.LocalRegion.checkRegionDestroyed(LocalRegion.java:8118) > at > com.gemstone.gemfire.internal.cache.LocalRegion.checkReadiness(LocalRegion.java:2994) > at > com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet$KeysSetIterator.next(PartitionedRegion.java:6544) > at > com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet.toArray(PartitionedRegion.java:6638) > at > com.gemstone.gemfire.internal.cache.PartitionedRegion$KeysSet.toArray(PartitionedRegion.java:6631) > at > java.util.Collections$UnmodifiableCollection.toArray(Collections.java:1033) > at java.util.ArrayList.addAll(ArrayList.java:577) > at > newWan.WANOperationsClient.getExistingKeys(WANOperationsClient.java:792) > at newWan.WANOperationsClient.putAll(WANOperationsClient.java:463) > at newWan.WANOperationsClient.doOps(WANOperationsClient.java:1419) > at newWan.WANTest.doOps(WANTest.java:1650) > at newWan.WANTest.HydraTask_doOps(WANTest.java:1639) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at hydra.MethExecutor.execute(MethExecutor.java:198) > at hydra.MethExecutor.execute(MethExecutor.java:162) > at hydra.TestTask.execute(TestTask.java:195) > at hydra.RemoteTestModule$1.run(RemoteTestModule.java:216) > Caused by: com.gemstone.gemfire.cache.DiskAccessException: For DiskStore: > disk3: Fatal error from asynchronous flusher thread, caused by > java.lang.AssertionError: Version stamp should have a member at this point > for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a > (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; > rv0; ds=0; time=1452091668017};member=null) > at > com.gemstone.gemfire.internal.cache.DiskStoreImpl$FlusherThread.run(DiskStoreImpl.java:1812) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.AssertionError: Version stamp should have a member at > this point for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a > (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; > rv0; ds=0; time=1452091668017};member=null) > at com.gemstone.gemfire.internal.cache.Oplog.create(Oplog.java:3553) > at > com.gemstone.gemfire.internal.cache.PersistentOplogSet.create(PersistentOplogSet.java:189) > at > com.gemstone.gemfire.internal.cache.DiskStoreImpl.put(DiskStoreImpl.java:751) > at > com.gemstone.gemfire.internal.cache.DiskRegion.put(DiskRegion.java:381) > at > com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeBytesToDisk(DiskEntry.java:874) > at > com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeToDisk(DiskEntry.java:867) > at > com.gemstone.gemfire.internal.cache.DiskEntry$Helper.writeToDisk(DiskEntry.java:857) > at > com.gemstone.gemfire.internal.cache.DiskEntry$Helper.doAsyncFlush(DiskEntry.java:1742) > at > com.gemstone.gemfire.internal.cache.DiskStoreImpl$FlusherThread.run(DiskStoreImpl.java:1768) > ... 1 more > [CIRCULAR REFERENCE:com.gemstone.gemfire.cache.DiskAccessException: > For DiskStore: disk3: Fatal error from asynchronous flusher thread, caused by > java.lang.AssertionError: Version stamp should have a member at this point > for entry VersionedThinDiskLRURegionEntryHeapStringKey2@71d56a > (key=Object_36851; rawValue=VMCachedDeserializable@454500130; version={v0; > rv0; ds=0; time=1452091668017};member=null)] > To reproduce it, run following bt test: > rollingupgrade/newWan/parallel/garminHCT.conf > bridgeHostsPerSite=2 > bridgeThreadsPerVM=2 > bridgeVMsPerHost=4 > clientMem=2g > edgeHostsPerSite=2 > edgeThreadsPerVM=5 > edgeVMsPerHost=5 > locatorHostsPerSite=2 > locatorThreadsPerVM=1 > locatorVMsPerHost=1 > resultWaitSec=1200 > serverMem=12g > wanSites=2 > with local.conf: > newWan.WANOperationsClientPrms-randModValueForDebug = true; > hydra.HostPrms-hostNames = > fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 > w2-2013-lin-14 w2-2013-lin-15\", 1, 4)" ncf > fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 > w2-2013-lin-14 w2-2013-lin-15\", 1, 4)" ncf > fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 > w2-2013-lin-14 w2-2013-lin-15\", 1, 4)" ncf > ; > hydra.HostPrms-resourceDirBaseMapFileName = /home/xzhou/bin/hostdirs.prop; > hydra.RegionPrms-cacheListeners = none; > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)