[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157195#comment-16157195
]
Anoop Sam John commented on HBASE-15968:
Do we need? Or lets have some meta info in file which says abt mvcc format?
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156995#comment-16156995
]
Phil Yang commented on HBASE-15968:
---
No jira currently. It will be a new HFile format version?
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156876#comment-16156876
]
Anoop Sam John commented on HBASE-15968:
Any jira for changing the vlong for Cell mvcc to be long? I feel we should do
this now
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[ https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131067#comment-16131067 ] Hudson commented on HBASE-15968: FAILURE: Integrated in Jenkins build HBASE-14070.HLC #233 (See [https://builds.apache.org/job/HBASE-14070.HLC/233/]) HBASE-15968 New behavior of versions considering mvcc and ts rather than (yangzhe1991: rev 1ac4152b1952202f99e216265c9c046e09ecd02e) * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/IncludeAllCompactionQueryMatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/ScanQueryMatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/DeleteTracker.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/ColumnTracker.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/querymatcher/TestNewVersionBehaviorTracker.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/StripeCompactionScanQueryMatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/NormalUserScanQueryMatcher.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingMemStore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/UserScanQueryMatcher.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/querymatcher/TestUserScanQueryMatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/LegacyScanQueryMatcher.java * (edit) hbase-shell/src/main/ruby/hbase/admin.rb * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestCoprocessorScanPolicy.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithDeletes.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityNewVersionBehaivorTracker.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsOnNewVersionBehaviorTable.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ColumnFamilyDescriptor.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestNewVersionBehaviorFromClientSide.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ColumnFamilyDescriptorBuilder.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanInfo.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/querymatcher/TestCompactionScanQueryMatcher.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/NewVersionBehaviorTracker.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/MinorCompactionScanQueryMatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/CompactionScanQueryMatcher.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/DropDeletesCompactionScanQueryMatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/MajorCompactionScanQueryMatcher.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMajorCompaction.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java HBASE-15968 (addendum) revert unrelated PE changing (yangzhe1991: rev e1cd59bbc4c370375ad71aabfcc02b16c46e5ba2) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > New behavior of versions considering mvcc and ts rather than ts only > > > Key: HBASE-15968 > URL: https://issues.apache.org/jira/browse/HBASE-15968 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch, > HBASE-15968.v07.patch,
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108532#comment-16108532
]
Anoop Sam John commented on HBASE-15968:
bq.If enable it by default, we may not use vlong in mvcc in HFile because we
will always keep them rather than set to 0.
+1. I came to ask this and saw ur comment.
May be we should think of changing this mvcc to be long only than vlong now
only? (2.0) In the past we used to reset the mvcc to be 0 immediately (no
concurrent scanners).. Later on after some jiras, we seems not reset this
fast. My thinking correct? So will it make sense to put it as long only? We
might need HFile version change then? Change from vlong to long will help us a
lot in the read path (Seek within an HFile block) and mainly for random gets.
cc [~stack]
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106461#comment-16106461
]
stack commented on HBASE-15968:
---
This is excellent.
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[ https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101006#comment-16101006 ] Hudson commented on HBASE-15968: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3434 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3434/]) HBASE-15968 New behavior of versions considering mvcc and ts rather than (yangzhe1991: rev 1ac4152b1952202f99e216265c9c046e09ecd02e) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/DropDeletesCompactionScanQueryMatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanInfo.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/ColumnTracker.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityNewVersionBehaivorTracker.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMajorCompaction.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithDeletes.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ColumnFamilyDescriptor.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestCoprocessorScanPolicy.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/MinorCompactionScanQueryMatcher.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/querymatcher/TestCompactionScanQueryMatcher.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestNewVersionBehaviorFromClientSide.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/querymatcher/TestNewVersionBehaviorTracker.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/NewVersionBehaviorTracker.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/DeleteTracker.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/NormalUserScanQueryMatcher.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/LegacyScanQueryMatcher.java * (edit) hbase-shell/src/main/ruby/hbase/admin.rb * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/MajorCompactionScanQueryMatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/StripeCompactionScanQueryMatcher.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/UserScanQueryMatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/CompactionScanQueryMatcher.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/querymatcher/TestUserScanQueryMatcher.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingMemStore.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ColumnFamilyDescriptorBuilder.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsOnNewVersionBehaviorTable.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/ScanQueryMatcher.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/IncludeAllCompactionQueryMatcher.java HBASE-15968 (addendum) revert unrelated PE changing (yangzhe1991: rev e1cd59bbc4c370375ad71aabfcc02b16c46e5ba2) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > New behavior of versions considering mvcc and ts rather than ts only > > > Key: HBASE-15968 > URL: https://issues.apache.org/jira/browse/HBASE-15968 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 3.0.0, 2.0.0-alpha-2 > > Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch, > HBASE-15968.v07.patch,
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100858#comment-16100858
]
Hudson commented on HBASE-15968:
FAILURE: Integrated in Jenkins build HBase-2.0 #233 (See
[https://builds.apache.org/job/HBase-2.0/233/])
HBASE-15968 New behavior of versions considering mvcc and ts rather than
(yangzhe1991: rev 86ce7fcbbf47758ed58da1a2ccbb0baf3cad1ac9)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ColumnFamilyDescriptor.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ColumnFamilyDescriptorBuilder.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanInfo.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/MajorCompactionScanQueryMatcher.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultCompactSelection.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsWithDeletes.java
* (edit) hbase-shell/src/main/ruby/hbase/admin.rb
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/ScanQueryMatcher.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/StripeCompactionScanQueryMatcher.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/ColumnTracker.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestNewVersionBehaviorFromClientSide.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/MinorCompactionScanQueryMatcher.java
* (add)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/IncludeAllCompactionQueryMatcher.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/DeleteTracker.java
* (add)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/NewVersionBehaviorTracker.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/DropDeletesCompactionScanQueryMatcher.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/LegacyScanQueryMatcher.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/NormalUserScanQueryMatcher.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestCoprocessorScanPolicy.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingMemStore.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMajorCompaction.java
* (add)
hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityNewVersionBehaivorTracker.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/UserScanQueryMatcher.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/querymatcher/TestNewVersionBehaviorTracker.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/CompactionScanQueryMatcher.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/querymatcher/TestUserScanQueryMatcher.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/querymatcher/TestCompactionScanQueryMatcher.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsOnNewVersionBehaviorTable.java
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099612#comment-16099612
]
Duo Zhang commented on HBASE-15968:
---
OK. +1.
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099611#comment-16099611
]
Phil Yang commented on HBASE-15968:
---
They are not related and I'll remove them when commit
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099603#comment-16099603
]
Duo Zhang commented on HBASE-15968:
---
Have you seem my comments on rb? Are the PE modifications related? Thanks.
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099595#comment-16099595
]
Phil Yang commented on HBASE-15968:
---
Checked two ruby tests, not fully understand the meaning.. But TestShell passed
and it only add a new attribute so I think it is OK. Will push to master and
branch-2 shortly.
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099348#comment-16099348
]
Duo Zhang commented on HBASE-15968:
---
Overall LGTM. +1.
Rubocup and ruby-lint? New pre commit check stuffs?
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097923#comment-16097923
]
Phil Yang commented on HBASE-15968:
---
Uploaded latest patch on https://reviews.apache.org/r/61070/
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968.v07.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096246#comment-16096246
]
Hadoop QA commented on HBASE-15968:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
6s{color} | {color:red} The patch generated 6 new + 322 unchanged - 5 fixed =
328 total (was 327) {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m
3s{color} | {color:red} The patch generated 11 new + 715 unchanged - 0 fixed =
726 total (was 715) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 59s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
40s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}113m
44s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
0s{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-15968 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12878335/HBASE-15968.v07.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile rubocop ruby_lint |
| uname | Linux c83927e815c0 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096089#comment-16096089
]
Hadoop QA commented on HBASE-15968:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
6s{color} | {color:red} The patch generated 6 new + 322 unchanged - 5 fixed =
328 total (was 327) {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m
2s{color} | {color:red} The patch generated 11 new + 715 unchanged - 0 fixed =
726 total (was 715) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 0s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
42s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 24s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
18s{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestDisableTableProcedure |
| | org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay |
| | org.apache.hadoop.hbase.master.procedure.TestModifyTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestCreateTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestEnableTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestDeleteTableProcedure |
| |
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096069#comment-16096069
]
Phil Yang commented on HBASE-15968:
---
Checked two failing tests, in current head of master TestShell also failed.
TestBlockCacheReporting always passed locally...
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968.v06.patch,
> HBASE-15968.v07.patch, HBASE-15968-v1.patch, HBASE-15968-v2.patch,
> HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094566#comment-16094566
]
Hadoop QA commented on HBASE-15968:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m
0s{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m
0s{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 20s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
39s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 38s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 28s{color}
| {color:red} hbase-shell in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.io.hfile.TestBlockCacheReporting |
| | hadoop.hbase.client.TestShell |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Issue | HBASE-15968 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12878140/HBASE-15968.v06.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile rubocop ruby_lint |
| uname | Linux 832de49dbdf4 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094453#comment-16094453
]
Hadoop QA commented on HBASE-15968:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m
0s{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m
0s{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 59s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
27s{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed =
1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
53s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 24m 39s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 32s{color}
| {color:red} hbase-shell in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Switch statement found in
org.apache.hadoop.hbase.security.visibility.VisibilityNewVersionBehaivorTracker.add(Cell)
where default case is missing At
VisibilityNewVersionBehaivorTracker.java:where default case is missing At
VisibilityNewVersionBehaivorTracker.java:[lines 121-139] |
| Failed junit tests | hadoop.hbase.io.hfile.TestBlockCacheReporting |
| | hadoop.hbase.client.TestShell |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:757bf37 |
| JIRA Issue | HBASE-15968 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094433#comment-16094433
]
Phil Yang commented on HBASE-15968:
---
This patch is for master, not sure when we can push to branch-2 and versions of
branch-2 can not be determined now (alpha-2 or alpha-3 or beta-xx..)
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094432#comment-16094432
]
Duo Zhang commented on HBASE-15968:
---
[~stack] I think we can commit this into 2.0 but should not enable it by
default. It must be used together with serial replication otherwise it may
cause other confusing behavior but serial replication has its own problem...
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094426#comment-16094426
]
stack commented on HBASE-15968:
---
You don't think this could make 2.0.0 [~yangzhe1991] ?
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 3.0.0
>
> Attachments: HBASE-15968.v05.patch, HBASE-15968-v1.patch,
> HBASE-15968-v2.patch, HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-15968) New behavior of versions considering mvcc and ts rather than ts only
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094298#comment-16094298
]
Hadoop QA commented on HBASE-15968:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HBASE-15968 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.4.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-15968 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829792/HBASE-15968-v4.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/7728/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
> New behavior of versions considering mvcc and ts rather than ts only
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch, HBASE-15968-v2.patch,
> HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable
