[jira] [Commented] (GEODE-7085) Cannot recover from disk store if region version is greater than Integer.MAX_VALUE
[ https://issues.apache.org/jira/browse/GEODE-7085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916882#comment-16916882 ] ASF subversion and git services commented on GEODE-7085: Commit 36ca311610d5b2a6950441cd75ce686f171b60c6 in geode's branch refs/heads/develop from Dan Smith [ https://gitbox.apache.org/repos/asf?p=geode.git;h=36ca311 ] GEODE-7085: Add dunit test of recovery with large versions (#3940) Adding a dunit test that we can recover from disk store files with a large gc version. > Cannot recover from disk store if region version is greater than > Integer.MAX_VALUE > -- > > Key: GEODE-7085 > URL: https://issues.apache.org/jira/browse/GEODE-7085 > Project: Geode > Issue Type: Bug > Components: membership, persistence >Reporter: Dan Smith >Assignee: Dan Smith >Priority: Major > Fix For: 1.10.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > > We hit an issue where a member failed to recover due to a > IndexOutOfBoundsException while recording a version during recovery. > Looking closer, it looks like the issue is due to the fact that a > RegionVersionHolder cannot record a version greater than Integer.MAX_VALUE if > it just just constructed. > When we are recovering from disk, the first thing we read from is the .drf > files. The first thing in those drf files is RVV information. We read the RVV > records and call recordRecoveredGCVersion. > When that call gets down inside RegionVersionHolder.recordVersion, there is > some logic that is supposed to flush out the bitSet and advance the > bitSetVersion. Unfortunately it looks like flushBitSetDuringRecording is not > actually doing that. So if version we read from disk is greater than > Integer.MAX_VALUE, we wrap around and try to set a negative index in the > bitset. > I can reproduce this with a unit test of RegionVersionVector that records a > version greater than Integer.MAX_VALUE. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (GEODE-7085) Cannot recover from disk store if region version is greater than Integer.MAX_VALUE
[ https://issues.apache.org/jira/browse/GEODE-7085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916007#comment-16916007 ] ASF subversion and git services commented on GEODE-7085: Commit 29120def0c006344b5cd3f15ae9d72d1a84b4566 in geode's branch refs/heads/release/1.10.0 from Dan Smith [ https://gitbox.apache.org/repos/asf?p=geode.git;h=29120de ] GEODE-7085: Ensuring the bitset stays within BIT_SET_WIDTH (#3922) Ensuring that when we call recordVersion on a RegionVersionHolder, we appropriately move the bitSet to match the new version we are recording, rather than trying to expand it. In particular, if new version is greater than Integer.MAX_VALUE, we can't record than in out integer indexed bit set. This change rewrites addBitSetExceptions. The logic is now broken into a BitSetExceptionIterator, which converts some or all of the bit set into RVVException objects, and the logic to slide the bit set forward to a new bitSetVersion. Adding unit tests that show that large versions cause an IndexOutOfBounds exception from recordGCVersion. Adding more unit tests for the internal state of the bitset. (cherry picked from commit f58710116db1cd8c509b59a43ffa050a073234d7) > Cannot recover from disk store if region version is greater than > Integer.MAX_VALUE > -- > > Key: GEODE-7085 > URL: https://issues.apache.org/jira/browse/GEODE-7085 > Project: Geode > Issue Type: Bug > Components: membership, persistence >Reporter: Dan Smith >Assignee: Dan Smith >Priority: Major > Fix For: 1.10.0 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > We hit an issue where a member failed to recover due to a > IndexOutOfBoundsException while recording a version during recovery. > Looking closer, it looks like the issue is due to the fact that a > RegionVersionHolder cannot record a version greater than Integer.MAX_VALUE if > it just just constructed. > When we are recovering from disk, the first thing we read from is the .drf > files. The first thing in those drf files is RVV information. We read the RVV > records and call recordRecoveredGCVersion. > When that call gets down inside RegionVersionHolder.recordVersion, there is > some logic that is supposed to flush out the bitSet and advance the > bitSetVersion. Unfortunately it looks like flushBitSetDuringRecording is not > actually doing that. So if version we read from disk is greater than > Integer.MAX_VALUE, we wrap around and try to set a negative index in the > bitset. > I can reproduce this with a unit test of RegionVersionVector that records a > version greater than Integer.MAX_VALUE. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (GEODE-7085) Cannot recover from disk store if region version is greater than Integer.MAX_VALUE
[ https://issues.apache.org/jira/browse/GEODE-7085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916008#comment-16916008 ] ASF subversion and git services commented on GEODE-7085: Commit e341c210005d0925b686c00f29b80e50b132ef5a in geode's branch refs/heads/release/1.10.0 from Dan Smith [ https://gitbox.apache.org/repos/asf?p=geode.git;h=e341c21 ] GEODE-7085: Ensure bitset is flushed in all code paths In recordVersion, there was a code path where we would not call flushBitSetDuringRecording before calling setVersionInBitSet. Moving the flush to the top of the method to ensure that the bit set is always flushed, and we never end up with a case where we try to set too large of a bit. By code inspection, we determined that initializeFrom was what got us into the state that triggered this code path - it could leave bitSetVersion at a small number while setting version to a large number. Fixing initializeFrom so that it correctly sets bitSetVersion. Co-Authored-By: Ernest Burghardt (cherry picked from commit f17931bf541fc0255112f713931388d9ee0bbc30) > Cannot recover from disk store if region version is greater than > Integer.MAX_VALUE > -- > > Key: GEODE-7085 > URL: https://issues.apache.org/jira/browse/GEODE-7085 > Project: Geode > Issue Type: Bug > Components: membership, persistence >Reporter: Dan Smith >Assignee: Dan Smith >Priority: Major > Fix For: 1.10.0 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > We hit an issue where a member failed to recover due to a > IndexOutOfBoundsException while recording a version during recovery. > Looking closer, it looks like the issue is due to the fact that a > RegionVersionHolder cannot record a version greater than Integer.MAX_VALUE if > it just just constructed. > When we are recovering from disk, the first thing we read from is the .drf > files. The first thing in those drf files is RVV information. We read the RVV > records and call recordRecoveredGCVersion. > When that call gets down inside RegionVersionHolder.recordVersion, there is > some logic that is supposed to flush out the bitSet and advance the > bitSetVersion. Unfortunately it looks like flushBitSetDuringRecording is not > actually doing that. So if version we read from disk is greater than > Integer.MAX_VALUE, we wrap around and try to set a negative index in the > bitset. > I can reproduce this with a unit test of RegionVersionVector that records a > version greater than Integer.MAX_VALUE. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (GEODE-7085) Cannot recover from disk store if region version is greater than Integer.MAX_VALUE
[ https://issues.apache.org/jira/browse/GEODE-7085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911720#comment-16911720 ] ASF subversion and git services commented on GEODE-7085: Commit f17931bf541fc0255112f713931388d9ee0bbc30 in geode's branch refs/heads/develop from Dan Smith [ https://gitbox.apache.org/repos/asf?p=geode.git;h=f17931b ] GEODE-7085: Ensure bitset is flushed in all code paths In recordVersion, there was a code path where we would not call flushBitSetDuringRecording before calling setVersionInBitSet. Moving the flush to the top of the method to ensure that the bit set is always flushed, and we never end up with a case where we try to set too large of a bit. By code inspection, we determined that initializeFrom was what got us into the state that triggered this code path - it could leave bitSetVersion at a small number while setting version to a large number. Fixing initializeFrom so that it correctly sets bitSetVersion. Co-Authored-By: Ernest Burghardt > Cannot recover from disk store if region version is greater than > Integer.MAX_VALUE > -- > > Key: GEODE-7085 > URL: https://issues.apache.org/jira/browse/GEODE-7085 > Project: Geode > Issue Type: Bug > Components: membership, persistence >Reporter: Dan Smith >Assignee: Dan Smith >Priority: Major > Fix For: 1.11.0 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > We hit an issue where a member failed to recover due to a > IndexOutOfBoundsException while recording a version during recovery. > Looking closer, it looks like the issue is due to the fact that a > RegionVersionHolder cannot record a version greater than Integer.MAX_VALUE if > it just just constructed. > When we are recovering from disk, the first thing we read from is the .drf > files. The first thing in those drf files is RVV information. We read the RVV > records and call recordRecoveredGCVersion. > When that call gets down inside RegionVersionHolder.recordVersion, there is > some logic that is supposed to flush out the bitSet and advance the > bitSetVersion. Unfortunately it looks like flushBitSetDuringRecording is not > actually doing that. So if version we read from disk is greater than > Integer.MAX_VALUE, we wrap around and try to set a negative index in the > bitset. > I can reproduce this with a unit test of RegionVersionVector that records a > version greater than Integer.MAX_VALUE. I’m looking into how to fix the > flushBitSetDuringRecording method. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (GEODE-7085) Cannot recover from disk store if region version is greater than Integer.MAX_VALUE
[ https://issues.apache.org/jira/browse/GEODE-7085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909270#comment-16909270 ] ASF subversion and git services commented on GEODE-7085: Commit f58710116db1cd8c509b59a43ffa050a073234d7 in geode's branch refs/heads/feature/GEODE-7066 from Dan Smith [ https://gitbox.apache.org/repos/asf?p=geode.git;h=f587101 ] GEODE-7085: Ensuring the bitset stays within BIT_SET_WIDTH (#3922) Ensuring that when we call recordVersion on a RegionVersionHolder, we appropriately move the bitSet to match the new version we are recording, rather than trying to expand it. In particular, if new version is greater than Integer.MAX_VALUE, we can't record than in out integer indexed bit set. This change rewrites addBitSetExceptions. The logic is now broken into a BitSetExceptionIterator, which converts some or all of the bit set into RVVException objects, and the logic to slide the bit set forward to a new bitSetVersion. Adding unit tests that show that large versions cause an IndexOutOfBounds exception from recordGCVersion. Adding more unit tests for the internal state of the bitset. > Cannot recover from disk store if region version is greater than > Integer.MAX_VALUE > -- > > Key: GEODE-7085 > URL: https://issues.apache.org/jira/browse/GEODE-7085 > Project: Geode > Issue Type: Bug > Components: membership, persistence >Reporter: Dan Smith >Assignee: Dan Smith >Priority: Major > Fix For: 1.11.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > We hit an issue where a member failed to recover due to a > IndexOutOfBoundsException while recording a version during recovery. > Looking closer, it looks like the issue is due to the fact that a > RegionVersionHolder cannot record a version greater than Integer.MAX_VALUE if > it just just constructed. > When we are recovering from disk, the first thing we read from is the .drf > files. The first thing in those drf files is RVV information. We read the RVV > records and call recordRecoveredGCVersion. > When that call gets down inside RegionVersionHolder.recordVersion, there is > some logic that is supposed to flush out the bitSet and advance the > bitSetVersion. Unfortunately it looks like flushBitSetDuringRecording is not > actually doing that. So if version we read from disk is greater than > Integer.MAX_VALUE, we wrap around and try to set a negative index in the > bitset. > I can reproduce this with a unit test of RegionVersionVector that records a > version greater than Integer.MAX_VALUE. I’m looking into how to fix the > flushBitSetDuringRecording method. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (GEODE-7085) Cannot recover from disk store if region version is greater than Integer.MAX_VALUE
[ https://issues.apache.org/jira/browse/GEODE-7085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908303#comment-16908303 ] ASF subversion and git services commented on GEODE-7085: Commit f58710116db1cd8c509b59a43ffa050a073234d7 in geode's branch refs/heads/develop from Dan Smith [ https://gitbox.apache.org/repos/asf?p=geode.git;h=f587101 ] GEODE-7085: Ensuring the bitset stays within BIT_SET_WIDTH (#3922) Ensuring that when we call recordVersion on a RegionVersionHolder, we appropriately move the bitSet to match the new version we are recording, rather than trying to expand it. In particular, if new version is greater than Integer.MAX_VALUE, we can't record than in out integer indexed bit set. This change rewrites addBitSetExceptions. The logic is now broken into a BitSetExceptionIterator, which converts some or all of the bit set into RVVException objects, and the logic to slide the bit set forward to a new bitSetVersion. Adding unit tests that show that large versions cause an IndexOutOfBounds exception from recordGCVersion. Adding more unit tests for the internal state of the bitset. > Cannot recover from disk store if region version is greater than > Integer.MAX_VALUE > -- > > Key: GEODE-7085 > URL: https://issues.apache.org/jira/browse/GEODE-7085 > Project: Geode > Issue Type: Bug > Components: membership, persistence >Reporter: Dan Smith >Assignee: Dan Smith >Priority: Major > Fix For: 1.11.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > We hit an issue where a member failed to recover due to a > IndexOutOfBoundsException while recording a version during recovery. > Looking closer, it looks like the issue is due to the fact that a > RegionVersionHolder cannot record a version greater than Integer.MAX_VALUE if > it just just constructed. > When we are recovering from disk, the first thing we read from is the .drf > files. The first thing in those drf files is RVV information. We read the RVV > records and call recordRecoveredGCVersion. > When that call gets down inside RegionVersionHolder.recordVersion, there is > some logic that is supposed to flush out the bitSet and advance the > bitSetVersion. Unfortunately it looks like flushBitSetDuringRecording is not > actually doing that. So if version we read from disk is greater than > Integer.MAX_VALUE, we wrap around and try to set a negative index in the > bitset. > I can reproduce this with a unit test of RegionVersionVector that records a > version greater than Integer.MAX_VALUE. I’m looking into how to fix the > flushBitSetDuringRecording method. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (GEODE-7085) Cannot recover from disk store if region version is greater than Integer.MAX_VALUE
[
https://issues.apache.org/jira/browse/GEODE-7085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906718#comment-16906718
]
Dan Smith commented on GEODE-7085:
--
Here is a stack trace from the unit test
{code}
@Test
public void recordGCVersionLargerThanIntMaxValueShouldSucceed() {
RegionVersionHolder h = createHolder(true);
long version = ((long) Integer.MAX_VALUE) + 10L;
h.recordVersion(version);
assertContainsOnly(h, version);
}
{code}
{noformat}
java.lang.IndexOutOfBoundsException: bitIndex < 0: -2147483639
at java.util.BitSet.set(BitSet.java:444)
at
org.apache.geode.internal.cache.versions.RegionVersionHolder.setVersionInBitSet(RegionVersionHolder.java:389)
at
org.apache.geode.internal.cache.versions.RegionVersionHolder.recordVersionWithBitSet(RegionVersionHolder.java:380)
at
org.apache.geode.internal.cache.versions.RegionVersionHolder.recordVersion(RegionVersionHolder.java:346)
at
org.apache.geode.internal.cache.versions.RegionVersionHolder2JUnitTest.recordGCVersionLargerThanIntMaxValueShouldSucceed(RegionVersionHolder2JUnitTest.java:167)
{noformat}
> Cannot recover from disk store if region version is greater than
> Integer.MAX_VALUE
> --
>
> Key: GEODE-7085
> URL: https://issues.apache.org/jira/browse/GEODE-7085
> Project: Geode
> Issue Type: Bug
> Components: membership, persistence
>Reporter: Dan Smith
>Assignee: Dan Smith
>Priority: Major
>
> We hit an issue where a member failed to recover due to a
> IndexOutOfBoundsException while recording a version during recovery.
> Looking closer, it looks like the issue is due to the fact that a
> RegionVersionHolder cannot record a version greater than Integer.MAX_VALUE if
> it just just constructed.
> When we are recovering from disk, the first thing we read from is the .drf
> files. The first thing in those drf files is RVV information. We read the RVV
> records and call recordRecoveredGCVersion.
> When that call gets down inside RegionVersionHolder.recordVersion, there is
> some logic that is supposed to flush out the bitSet and advance the
> bitSetVersion. Unfortunately it looks like flushBitSetDuringRecording is not
> actually doing that. So if version we read from disk is greater than
> Integer.MAX_VALUE, we wrap around and try to set a negative index in the
> bitset.
> I can reproduce this with a unit test of RegionVersionVector that records a
> version greater than Integer.MAX_VALUE. I’m looking into how to fix the
> flushBitSetDuringRecording method.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
