[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5010:
--
Comment: was deleted
(was: Integrated in HBase-TRUNK-security #129 (See
[https://builds.apache.org/job/HBase-TRUNK-security/129/])
HBASE-5010 Pass region info in
LoadBalancer.randomAssignment(ListServerName servers) (Anoop Sam John)
(Revision 1297155)
Result = FAILURE
tedyu :
Files :
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
)
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Fix For: 0.94.0
Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
D909.2.patch, D909.3.patch, D909.4.patch, D909.5.patch, D909.6.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5010:
--
Comment: was deleted
(was: Integrated in HBase-TRUNK #2672 (See
[https://builds.apache.org/job/HBase-TRUNK/2672/])
HBASE-5010 Pass region info in
LoadBalancer.randomAssignment(ListServerName servers) (Anoop Sam John)
(Revision 1297155)
Result = SUCCESS
tedyu :
Files :
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
)
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Fix For: 0.94.0
Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
D909.2.patch, D909.3.patch, D909.4.patch, D909.5.patch, D909.6.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Bautin updated HBASE-5010:
--
Resolution: Fixed
Fix Version/s: 0.94.0
Assignee: Mikhail Bautin (was: Zhihong Yu)
Status: Resolved (was: Patch Available)
A follow-up fix was submitted as part of HBASE-5274 to bring the trunk fix for
this issue to parity with the 89-fb fix. Resolving.
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Fix For: 0.94.0
Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
D909.2.patch, D909.3.patch, D909.4.patch, D909.5.patch, D909.6.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HBASE-5010:
---
Attachment: D909.6.patch
mbautin updated the revision [jira] [HBASE-5010] [89-fb] Filter HFiles based
on TTL.
Reviewers: Kannan, Liyin, JIRA
Addressing the issue that Kannan pointed out: getScanner does not use its
arguments. It turned out that getScanner was called with this.scan and
this.columns in all callsites, so I have removed those arguments.
REVISION DETAIL
https://reviews.facebook.net/D909
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
src/main/java/org/apache/hadoop/hbase/util/Threads.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Zhihong Yu
Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
D909.2.patch, D909.3.patch, D909.4.patch, D909.5.patch, D909.6.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HBASE-5010:
---
Attachment: D909.4.patch
mbautin updated the revision [jira] [HBASE-5010] [89-fb] Filter HFiles based
on TTL.
Reviewers: Kannan, Liyin, JIRA
Addressing Kannan's comment about doing the same optimization during
compactions. Adding a compaction test to the unit test, and verifying that we
don't read expired files using per-CF metrics.
REVISION DETAIL
https://reviews.facebook.net/D909
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
src/main/java/org/apache/hadoop/hbase/util/Threads.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Zhihong Yu
Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
D909.2.patch, D909.3.patch, D909.4.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HBASE-5010:
---
Attachment: D909.5.patch
mbautin updated the revision [jira] [HBASE-5010] [89-fb] Filter HFiles based
on TTL.
Reviewers: Kannan, Liyin, JIRA
Actually, this is where I addressed Kannan's comment about compactions:
https://reviews.facebook.net/D909?vs=2679id=3015whitespace=ignore-all
(see the new line 154 in StoreScanner.java:
scanners = selectScannersFrom(scanners);
)
I have spent quite a bit of time making sure that the unit test is testing
the optimization during compactions. I am now invoking compactions in two
different ways, for a total of 6 parameterized instances of the test, but it is
still really quick. All of our compaction codepaths go through StoreScanner
constructors, so we should have the optimization in all of them.
All unit tests pass.
REVISION DETAIL
https://reviews.facebook.net/D909
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java
src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java
src/main/java/org/apache/hadoop/hbase/util/Threads.java
src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Zhihong Yu
Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
D909.2.patch, D909.3.patch, D909.4.patch, D909.5.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HBASE-5010:
---
Attachment: D1017.2.patch
mbautin updated the revision [jira] [HBASE-5010] Filter HFiles based on TTL.
Reviewers: Kannan, Liyin, tedyu, JIRA
Addressing Ted's comment and getting rid of unused ttl parameters in a couple
of places.
REVISION DETAIL
https://reviews.facebook.net/D1017
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestSelectScannersUsingTTL.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: D1017.1.patch, D1017.2.patch, D909.1.patch, D909.2.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5010:
--
Comment: was deleted
(was: -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12508625/D1017.2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 29 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/599//console
This message is automatically generated.)
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: D1017.1.patch, D1017.2.patch, D909.1.patch, D909.2.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5010:
--
Attachment: 5010.patch
Moved the new test to
src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java
Added javadoc for oldestUnexpiredTS
Added test category for TestScannerSelectionUsingTTL
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
D909.2.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5010:
--
Status: Patch Available (was: Open)
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
D909.2.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5010:
--
Status: Open (was: Patch Available)
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
D909.2.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5010:
--
Status: Patch Available (was: Open)
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
D909.2.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5010:
--
Status: Open (was: Patch Available)
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
D909.2.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5010:
--
Attachment: (was: 5010.patch)
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
D909.2.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5010:
--
Attachment: 5010.patch
Fix compilation error in TestScannerSelectionUsingTTL
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
D909.2.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5010:
--
Status: Open (was: Patch Available)
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
D909.2.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5010:
--
Status: Patch Available (was: Open)
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
D909.2.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HBASE-5010:
---
Attachment: D1017.1.patch
mbautin requested code review of [jira] [HBASE-5010] Filter HFiles based on
TTL.
Reviewers: Kannan, Liyin, tedyu, JIRA
This is the trunk version of D909. The main difference is that there is a
minVersions CF setting in trunk, and when minVersions is not zero, we can't
exclude StoreFiles based on TTL, because we might have to retrieve KVs with
expired timestamps to comply with the minVersions requirement.
TEST PLAN
Unit tests (including a new one).
REVISION DETAIL
https://reviews.facebook.net/D1017
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestSelectScannersUsingTTL.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/2127/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: D1017.1.patch, D909.1.patch, D909.2.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5010:
--
Status: Patch Available (was: Open)
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: D1017.1.patch, D909.1.patch, D909.2.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HBASE-5010:
---
Attachment: D909.2.patch
mbautin updated the revision [jira] [HBASE-5010] [89-fb] Filter HFiles based
on TTL.
Reviewers: Kannan, Liyin, JIRA
Addressing Kannan's and Ted's comments.
REVISION DETAIL
https://reviews.facebook.net/D909
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java
src/main/java/org/apache/hadoop/hbase/regionserver/NonLazyKeyValueScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java
src/main/java/org/apache/hadoop/hbase/util/Threads.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestSelectScannersUsingTTL.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Attachments: D909.1.patch, D909.2.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Bautin updated HBASE-5010:
--
Description:
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
was:
In ScanWildcardColumnTracker we have
{
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
