[
https://issues.apache.org/jira/browse/GEODE-92?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anilkumar Gingade updated GEODE-92:
-----------------------------------
Description:
I discovered this while working on a unit test for another issue. It appears
that when configuring a PR with entry eviction 1, in some cases there ends up
being more than 1 entry left in memory when using overflow to disk with
disk-synchronous=false.
With synchronous disk, this issue does not occur.
A deadlock was also found when doing a put on a region with custom expiry
callback doing getValue():
"ServerConnection on port 40404 Thread 1543350" tid=0x5267ad owned by
"ServerConnection on port 40404 Thread 1543342" tid=0x52679b
java.lang.Thread.State: BLOCKED
at
org.apache.geode.internal.cache.eviction.AbstractEvictionList.isEvictable(AbstractEvictionList.java:199)
- blocked on
org.apache.geode.internal.cache.entries.VersionedStatsLRURegionEntryHeapObjectKey@6027657a
at
org.apache.geode.internal.cache.eviction.LRUListWithSyncSorting.getEvictableEntry(LRUListWithSyncSorting.java:72)
at
org.apache.geode.internal.cache.AbstractLRURegionMap.lruUpdateCallback(AbstractLRURegionMap.java:445)
at
org.apache.geode.internal.cache.LocalRegion.getDeserialized(LocalRegion.java:1276)
at
org.apache.geode.internal.cache.LocalRegion$ExpiryRegionEntry.getValue(LocalRegion.java:7893)
at
org.apache.geode.modules.util.SessionCustomExpiry.getExpiry(SessionCustomExpiry.java:38)
at
org.apache.geode.internal.cache.LocalRegion.createExpiryTask(LocalRegion.java:7995)
at
org.apache.geode.internal.cache.LocalRegion.addExpiryTask(LocalRegion.java:8109)
at
org.apache.geode.internal.cache.LocalRegion.addExpiryTaskIfAbsent(LocalRegion.java:7810)
at
org.apache.geode.internal.cache.LocalRegion.updateStatsForPut(LocalRegion.java:7116)
at
org.apache.geode.internal.cache.LocalRegion.basicPutPart2(LocalRegion.java:5739)
at
org.apache.geode.internal.cache.BucketRegion.basicPutPart2(BucketRegion.java:677)
at
org.apache.geode.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:2864)
- locked
org.apache.geode.internal.cache.entries.VersionedStatsLRURegionEntryHeapObjectKey@4efa554
at
org.apache.geode.internal.cache.BucketRegion.virtualPut(BucketRegion.java:503)
at
org.apache.geode.internal.cache.PartitionedRegionDataStore.putLocally(PartitionedRegionDataStore.java:1223)
at
org.apache.geode.internal.cache.PartitionedRegion.putInBucket(PartitionedRegion.java:2819)
at
org.apache.geode.internal.cache.PartitionedRegion.virtualPut(PartitionedRegion.java:2027)
at
org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:152)
at
org.apache.geode.internal.cache.LocalRegion.basicUpdate(LocalRegion.java:5603)
at
org.apache.geode.internal.cache.LocalRegion.basicBridgePut(LocalRegion.java:5240)
at
org.apache.geode.internal.cache.tier.sockets.command.Put65.cmdExecute(Put65.java:390)
at
org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:163)
at
org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:797)
at
org.apache.geode.internal.cache.tier.sockets.LegacyServerConnection.doOneMessage(LegacyServerConnection.java:85)
at
org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1179)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$4$1.run(AcceptorImpl.java:641)
at java.lang.Thread.run(Thread.java:748)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@b5c77c6
was:
I discovered this while working on a unit test for another issue. It appears
that when configuring a PR with entry eviction 1, in some cases there ends up
being more than 1 entry left in memory when using overflow to disk with
disk-synchronous=false.
With synchronous disk, this issue does not occur.
A deadlock was also found when doing a put on a region with custom expiry
callback doing getValue().
h2. Activity
* *All*
* *Comments*
* *Work Log*
* *History*
* *Activity*
* *Transitions*
Ascending order - Click to sort in descending order
Permalink
[~upthewaterspout] added a comment - 10/Jul/15 18:31
Unit test that demostrates this issue.
Permalink
[~jira-bot] added a comment - 14/Jul/15 21:22
Commit dceac32e8ec60c3c7d32324cabd74f1ae5451fac in incubator-geode's branch
refs/heads/feature/GEODE-56 from [~upthewaterspout]
[ [https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=dceac32] ]
GEODE-56 Decrementing bytesOverflowedOnDisk when an update happens to an
evicted entry.
For persistent regions, we were correctly updating this stat. But for
non persistent regions we did not read the old size on disk, so we were
not updating the stat.
If the entry was in the async queue, we only increment the stat once
when the final value is actually written to disk. So we will only
decrement the stat if the oldValue is not pending asynchronous write to
disk.
I'm still seeing issues with the entryCount. I've tracked this down to
what I think is a problem with tombstones. It appears when we modify an
entry that is a tombstone, we increment the entry count once for the new
value, and once for removing the tombstone. I've filed GEODE-92 and
GEODE-93 for the issues I've found with this new test.
Permalink
[~jira-bot] added a comment - 21/Jul/15 21:47
Commit b8686b52b97f8cbe1b46df8cda81a5e2bbd4cfaf in incubator-geode's branch
refs/heads/develop from [~upthewaterspout]
[ [https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=b8686b5] ]
GEODE-56 Decrementing bytesOverflowedOnDisk when an update happens to an
evicted entry.
For persistent regions, we were correctly updating this stat. But for
non persistent regions we did not read the old size on disk, so we were
not updating the stat.
If the entry was in the async queue, we only increment the stat once
when the final value is actually written to disk. So we will only
decrement the stat if the oldValue is not pending asynchronous write to
disk.
I'm still seeing issues with the entryCount. I've tracked this down to
what I think is a problem with tombstones. It appears when we modify an
entry that is a tombstone, we increment the entry count once for the new
value, and once for removing the tombstone. I've filed GEODE-92 and
GEODE-93 for the issues I've found with this new test.
Permalink
[~jira-bot] added a comment - 23/Jul/15 16:18
Commit b8686b52b97f8cbe1b46df8cda81a5e2bbd4cfaf in incubator-geode's branch
refs/heads/feature/GEODE-9 from [~upthewaterspout]
[ [https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=b8686b5] ]
GEODE-56 Decrementing bytesOverflowedOnDisk when an update happens to an
evicted entry.
For persistent regions, we were correctly updating this stat. But for
non persistent regions we did not read the old size on disk, so we were
not updating the stat.
If the entry was in the async queue, we only increment the stat once
when the final value is actually written to disk. So we will only
decrement the stat if the oldValue is not pending asynchronous write to
disk.
I'm still seeing issues with the entryCount. I've tracked this down to
what I think is a problem with tombstones. It appears when we modify an
entry that is a tombstone, we increment the entry count once for the new
value, and once for removing the tombstone. I've filed GEODE-92 and
GEODE-93 for the issues I've found with this new test.
Permalink
[~jira-bot] added a comment - 10/May/16 15:37
Commit f4b0643829b00d0ba9b0852fdd3a679812395545 in incubator-geode's branch
refs/heads/develop from [~sboorlagadda]
[ [https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=f4b0643] ]
GEODE-92: PR with entry eviction 1 leaves 3 entries in memory with async
overflow
Found that invoking lruUpdateCallback multiple times with in a single get
could mess up entry eviction state. So disabled lru callback and invoked
at the end of the operation.
Permalink
[~jira-bot] added a comment - 10/May/16 21:08
Commit f4b0643829b00d0ba9b0852fdd3a679812395545 in incubator-geode's branch
refs/heads/feature/GEODE-951 from [~sboorlagadda]
[ [https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=f4b0643] ]
GEODE-92: PR with entry eviction 1 leaves 3 entries in memory with async
overflow
Found that invoking lruUpdateCallback multiple times with in a single get
could mess up entry eviction state. So disabled lru callback and invoked
at the end of the operation.
Permalink
[~jira-bot] added a comment - 10/May/16 21:09
Commit f4b0643829b00d0ba9b0852fdd3a679812395545 in incubator-geode's branch
refs/heads/feature/GEODE-1371 from [~sboorlagadda]
[ [https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=f4b0643] ]
GEODE-92: PR with entry eviction 1 leaves 3 entries in memory with async
overflow
Found that invoking lruUpdateCallback multiple times with in a single get
could mess up entry eviction state. So disabled lru callback and invoked
at the end of the operation.
Permalink
[~jira-bot] added a comment - 10/May/16 21:11
Commit f4b0643829b00d0ba9b0852fdd3a679812395545 in incubator-geode's branch
refs/heads/feature/GEODE-1369 from [~sboorlagadda]
[ [https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=f4b0643] ]
GEODE-92: PR with entry eviction 1 leaves 3 entries in memory with async
overflow
Found that invoking lruUpdateCallback multiple times with in a single get
could mess up entry eviction state. So disabled lru callback and invoked
at the end of the operation.
Permalink
[~jira-bot] added a comment - 10/May/16 21:17
Commit f4b0643829b00d0ba9b0852fdd3a679812395545 in incubator-geode's branch
refs/heads/feature/GEODE-1370 from [~sboorlagadda]
[ [https://git-wip-us.apache.org/repos/asf?p=incubator-geode.git;h=f4b0643] ]
GEODE-92: PR with entry eviction 1 leaves 3 entries in memory with async
overflow
Found that invoking lruUpdateCallback multiple times with in a single get
could mess up entry eviction state. So disabled lru callback and invoked
at the end of the operation.
Permalink [
Edit|https://issues.apache.org/jira/secure/EditComment!default.jspa?id=12844242&commentId=16733693]
[
Delete|https://issues.apache.org/jira/secure/DeleteComment!default.jspa?id=12844242&commentId=16733693]
[~agingade] added a comment - 4 days ago
Re-opneing:
The commit f4b0643829b00d0ba9b0852fdd3a679812395545 introduced deadlock between
concurrent updates with custom expiry which invoked getValue() from the region
entry.
"ServerConnection on port 40404 Thread 1543350" tid=0x5267ad owned by
"ServerConnection on port 40404 Thread 1543342" tid=0x52679b
java.lang.Thread.State: BLOCKED
at
org.apache.geode.internal.cache.eviction.AbstractEvictionList.isEvictable(AbstractEvictionList.java:199)
- blocked on
org.apache.geode.internal.cache.entries.VersionedStatsLRURegionEntryHeapObjectKey@6027657a
at
org.apache.geode.internal.cache.eviction.LRUListWithSyncSorting.getEvictableEntry(LRUListWithSyncSorting.java:72)
at
org.apache.geode.internal.cache.AbstractLRURegionMap.lruUpdateCallback(AbstractLRURegionMap.java:445)
at
org.apache.geode.internal.cache.LocalRegion.getDeserialized(LocalRegion.java:1276)
at
org.apache.geode.internal.cache.LocalRegion$ExpiryRegionEntry.getValue(LocalRegion.java:7893)
at
org.apache.geode.modules.util.SessionCustomExpiry.getExpiry(SessionCustomExpiry.java:38)
at
org.apache.geode.internal.cache.LocalRegion.createExpiryTask(LocalRegion.java:7995)
at
org.apache.geode.internal.cache.LocalRegion.addExpiryTask(LocalRegion.java:8109)
at
org.apache.geode.internal.cache.LocalRegion.addExpiryTaskIfAbsent(LocalRegion.java:7810)
at
org.apache.geode.internal.cache.LocalRegion.updateStatsForPut(LocalRegion.java:7116)
at
org.apache.geode.internal.cache.LocalRegion.basicPutPart2(LocalRegion.java:5739)
at
org.apache.geode.internal.cache.BucketRegion.basicPutPart2(BucketRegion.java:677)
at
org.apache.geode.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:2864)
- locked
org.apache.geode.internal.cache.entries.VersionedStatsLRURegionEntryHeapObjectKey@4efa554
at
org.apache.geode.internal.cache.BucketRegion.virtualPut(BucketRegion.java:503)
at
org.apache.geode.internal.cache.PartitionedRegionDataStore.putLocally(PartitionedRegionDataStore.java:1223)
at
org.apache.geode.internal.cache.PartitionedRegion.putInBucket(PartitionedRegion.java:2819)
at
org.apache.geode.internal.cache.PartitionedRegion.virtualPut(PartitionedRegion.java:2027)
at
org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:152)
at
org.apache.geode.internal.cache.LocalRegion.basicUpdate(LocalRegion.java:5603)
at
org.apache.geode.internal.cache.LocalRegion.basicBridgePut(LocalRegion.java:5240)
at
org.apache.geode.internal.cache.tier.sockets.command.Put65.cmdExecute(Put65.java:390)
at
org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:163)
at
org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:797)
at
org.apache.geode.internal.cache.tier.sockets.LegacyServerConnection.doOneMessage(LegacyServerConnection.java:85)
at
org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1179)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$4$1.run(AcceptorImpl.java:641)
at java.lang.Thread.run(Thread.java:748)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@b5c77c6
> PR with entry eviction 1 leaves 3 entries in memory with async overflow
> -----------------------------------------------------------------------
>
> Key: GEODE-92
> URL: https://issues.apache.org/jira/browse/GEODE-92
> Project: Geode
> Issue Type: Bug
> Components: eviction
> Reporter: Dan Smith
> Assignee: Sai Boorlagadda
> Priority: Major
> Fix For: 1.0.0-incubating.M3
>
> Attachments:
> 0001-Test-that-demonstrates-two-many-entries-in-memory-wi.patch
>
>
> I discovered this while working on a unit test for another issue. It appears
> that when configuring a PR with entry eviction 1, in some cases there ends up
> being more than 1 entry left in memory when using overflow to disk with
> disk-synchronous=false.
> With synchronous disk, this issue does not occur.
> A deadlock was also found when doing a put on a region with custom expiry
> callback doing getValue():
> "ServerConnection on port 40404 Thread 1543350" tid=0x5267ad owned by
> "ServerConnection on port 40404 Thread 1543342" tid=0x52679b
> java.lang.Thread.State: BLOCKED
> at
> org.apache.geode.internal.cache.eviction.AbstractEvictionList.isEvictable(AbstractEvictionList.java:199)
> - blocked on
> org.apache.geode.internal.cache.entries.VersionedStatsLRURegionEntryHeapObjectKey@6027657a
> at
> org.apache.geode.internal.cache.eviction.LRUListWithSyncSorting.getEvictableEntry(LRUListWithSyncSorting.java:72)
> at
> org.apache.geode.internal.cache.AbstractLRURegionMap.lruUpdateCallback(AbstractLRURegionMap.java:445)
> at
> org.apache.geode.internal.cache.LocalRegion.getDeserialized(LocalRegion.java:1276)
> at
> org.apache.geode.internal.cache.LocalRegion$ExpiryRegionEntry.getValue(LocalRegion.java:7893)
> at
> org.apache.geode.modules.util.SessionCustomExpiry.getExpiry(SessionCustomExpiry.java:38)
> at
> org.apache.geode.internal.cache.LocalRegion.createExpiryTask(LocalRegion.java:7995)
> at
> org.apache.geode.internal.cache.LocalRegion.addExpiryTask(LocalRegion.java:8109)
> at
> org.apache.geode.internal.cache.LocalRegion.addExpiryTaskIfAbsent(LocalRegion.java:7810)
> at
> org.apache.geode.internal.cache.LocalRegion.updateStatsForPut(LocalRegion.java:7116)
> at
> org.apache.geode.internal.cache.LocalRegion.basicPutPart2(LocalRegion.java:5739)
> at
> org.apache.geode.internal.cache.BucketRegion.basicPutPart2(BucketRegion.java:677)
> at
> org.apache.geode.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:2864)
> - locked
> org.apache.geode.internal.cache.entries.VersionedStatsLRURegionEntryHeapObjectKey@4efa554
> at
> org.apache.geode.internal.cache.BucketRegion.virtualPut(BucketRegion.java:503)
> at
> org.apache.geode.internal.cache.PartitionedRegionDataStore.putLocally(PartitionedRegionDataStore.java:1223)
> at
> org.apache.geode.internal.cache.PartitionedRegion.putInBucket(PartitionedRegion.java:2819)
> at
> org.apache.geode.internal.cache.PartitionedRegion.virtualPut(PartitionedRegion.java:2027)
> at
> org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:152)
> at
> org.apache.geode.internal.cache.LocalRegion.basicUpdate(LocalRegion.java:5603)
> at
> org.apache.geode.internal.cache.LocalRegion.basicBridgePut(LocalRegion.java:5240)
> at
> org.apache.geode.internal.cache.tier.sockets.command.Put65.cmdExecute(Put65.java:390)
> at
> org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:163)
> at
> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:797)
> at
> org.apache.geode.internal.cache.tier.sockets.LegacyServerConnection.doOneMessage(LegacyServerConnection.java:85)
> at
> org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$4$1.run(AcceptorImpl.java:641)
> at java.lang.Thread.run(Thread.java:748)
> Number of locked synchronizers = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@b5c77c6
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
