Barrett Oglesby created GEODE-8827:
--------------------------------------
Summary: The DiskRegionStats bytesOnlyOnDisk stat is not
incremented during persistent region recovery
Key: GEODE-8827
URL: https://issues.apache.org/jira/browse/GEODE-8827
Project: Geode
Issue Type: Bug
Components: persistence, statistics
Reporter: Barrett Oglesby
With a test like:
- 2 servers with partitioned region:
- persistent
- heap eviction with overflow
- load enough entries to cause overflow
- shut down the servers
- restart the servers
- execute a function to get all entries in each server
After the step to restart the servers, the bytesOnlyOnDisk stat is 0.
After the step to get all entries, the bytesOnlyOnDisk stat is negative.
The entriesInVM and entriesOnlyOnDisk stats are incremented as BucketRegions
are recovered from disk in LocalRegion.initializeStats here:
{noformat}
java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(Thread.java:1333)
at
org.apache.geode.internal.cache.LocalRegion.initializeStats(LocalRegion.java:10222)
at
org.apache.geode.internal.cache.BucketRegion.initializeStats(BucketRegion.java:2163)
at
org.apache.geode.internal.cache.AbstractDiskRegion.copyExistingRegionMap(AbstractDiskRegion.java:775)
at
org.apache.geode.internal.cache.DiskStoreImpl.initializeOwner(DiskStoreImpl.java:631)
at
org.apache.geode.internal.cache.DiskRegion.initializeOwner(DiskRegion.java:239)
at
org.apache.geode.internal.cache.DistributedRegion.initialize(DistributedRegion.java:1081)
at
org.apache.geode.internal.cache.BucketRegion.initialize(BucketRegion.java:262)
at
org.apache.geode.internal.cache.LocalRegion.createSubregion(LocalRegion.java:981)
at
org.apache.geode.internal.cache.PartitionedRegionDataStore.createBucketRegion(PartitionedRegionDataStore.java:785)
at
org.apache.geode.internal.cache.PartitionedRegionDataStore.grabFreeBucket(PartitionedRegionDataStore.java:460)
at
org.apache.geode.internal.cache.PartitionedRegionDataStore.grabFreeBucketRecursively(PartitionedRegionDataStore.java:319)
at
org.apache.geode.internal.cache.PartitionedRegionDataStore.grabBucket(PartitionedRegionDataStore.java:2896)
at
org.apache.geode.internal.cache.ProxyBucketRegion.recoverFromDisk(ProxyBucketRegion.java:441)
at
org.apache.geode.internal.cache.ProxyBucketRegion.recoverFromDiskRecursively(ProxyBucketRegion.java:407)
at
org.apache.geode.internal.cache.PRHARedundancyProvider$2.run2(PRHARedundancyProvider.java:1640)
at
org.apache.geode.internal.cache.partitioned.RecoveryRunnable.run(RecoveryRunnable.java:60)
at
org.apache.geode.internal.cache.PRHARedundancyProvider$2.run(PRHARedundancyProvider.java:1630)
at java.lang.Thread.run(Thread.java:745)
{noformat}
The current LocalRegion.initializeStats method implementation is:
{noformat}
public void initializeStats(long numEntriesInVM, long numOverflowOnDisk,
long numOverflowBytesOnDisk) {
getDiskRegion().getStats().incNumEntriesInVM(numEntriesInVM);
getDiskRegion().getStats().incNumOverflowOnDisk(numOverflowOnDisk);
}
{noformat}
Even though numOverflowBytesOnDisk is passed into this method, it is ignored as
this logging shows:
{noformat}
[warn 2021/01/12 11:19:11.785 PST <Recovery thread for bucket _B__data_3>
tid=0x49] XXX LocalRegion.initializeStats numOverflowBytesOnDisk=4546560;
bytesOnlyOnDiskFromStats=0
[warn 2021/01/12 11:19:11.791 PST <Recovery thread for bucket _B__data_9>
tid=0x4f] XXX LocalRegion.initializeStats numOverflowBytesOnDisk=4536320;
bytesOnlyOnDiskFromStats=0
[warn 2021/01/12 11:19:11.797 PST <Recovery thread for bucket _B__data_6>
tid=0x4c] XXX LocalRegion.initializeStats numOverflowBytesOnDisk=4526080;
bytesOnlyOnDiskFromStats=0
[warn 2021/01/12 11:19:11.800 PST <Recovery thread for bucket _B__data_2>
tid=0x48] XXX LocalRegion.initializeStats numOverflowBytesOnDisk=4546560;
bytesOnlyOnDiskFromStats=0
[warn 2021/01/12 11:19:11.801 PST <Recovery thread for bucket _B__data_8>
tid=0x4e] XXX LocalRegion.initializeStats numOverflowBytesOnDisk=4526080;
bytesOnlyOnDiskFromStats=0
{noformat}
The attached
DiskRegionStats_entriesOnlyOnDisk_bytesOnlyOnDisk_after_restart.gif chart shows
the behavior after server restart.
The attached DiskRegionStats_entriesOnlyOnDisk_bytesOnlyOnDisk_after_gets.gif
chart shows the behavior after the gets.
If I make this change, I see correct behavior:
{noformat}
public void initializeStats(long numEntriesInVM, long numOverflowOnDisk,
long numOverflowBytesOnDisk) {
getDiskRegion().getStats().incNumEntriesInVM(numEntriesInVM);
getDiskRegion().getStats().incNumOverflowOnDisk(numOverflowOnDisk);
->
getDiskRegion().getStats().incNumOverflowBytesOnDisk(numOverflowBytesOnDisk);
}
{noformat}
The attached
DiskRegionStats_entriesOnlyOnDisk_bytesOnlyOnDisk_after_restart_with_change.gif
chart shows the behavior with this change after server restart.
The attached
DiskRegionStats_entriesOnlyOnDisk_bytesOnlyOnDisk_after_gets_with_change.gif
chart shows the behavior with this change after the gets.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
