[jira] [Commented] (HBASE-4215) RS requestsPerSecond counter seems to be off
[ https://issues.apache.org/jira/browse/HBASE-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088577#comment-13088577 ] subramanian raghunathan commented on HBASE-4215: 1) Modifying the requestPersecond value to integer rather than float since the value is used in several VO's like: 1) MasterMetrics metrics 2) AClusterStatus 3) StorageClusterStatusModel 4) AServerLoad etc. Also to maintain consistensy the regionservermetrics and HServerLoad using integer values for requestPersecond. Advantages: Consitency throughout the system . Changes are minimal. DisAdvantages: Percision loss, also integer can accomadate less value than the float. 2)HServerLoad.getNumberOfRequests() is used in multiple places. {color:green}StorageClusterStatusModel - Expects request per second ClusterStatus - Expects request per second MasterStatusTmplImpl - UI part also Expects request per second {color} {color:red}AvroUtil - Not sure {color} {color:red} HMaster public void regionServerReport(final byte [] sn, final HServerLoad hsl) throws IOException { this.serverManager.regionServerReport(new ServerName(sn), hsl); if (hsl != null this.metrics != null) { // Up our metrics. this.metrics.incrementRequests(hsl.getTotalNumberOfRequests()); } } The metric expects the total request in the all the region servers and the metric computes the per second value of the aggregation of all the region servers.{color} So following change is sugested to keep the existing attribute as the requestPersecond.Also introduce additional attribute that reports the total request made in RS as totalNumberOfRequests This will be used by the Master Metrics. Will upload the patch ASAP RS requestsPerSecond counter seems to be off Key: HBASE-4215 URL: https://issues.apache.org/jira/browse/HBASE-4215 Project: HBase Issue Type: Bug Components: metrics Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.92.0 In testing trunk, I had YCSB reporting some 40,000 requests/second, but the summary info on the master webpage was consistently indicating somewhere around 3x that. I'm guessing that we may have a bug where we forgot to divide by time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4229) Replace Jettison JSON encoding with Jackson in HLogPrettyPrinter
[ https://issues.apache.org/jira/browse/HBASE-4229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088592#comment-13088592 ] Hudson commented on HBASE-4229: --- Integrated in HBase-TRUNK #2127 (See [https://builds.apache.org/job/HBase-TRUNK/2127/]) HBASE-4229 Replace Jettison JSON encoding with Jackson in HLogPrettyPrinter stack : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogPrettyPrinter.java * /hbase/trunk/CHANGES.txt * /hbase/trunk/bin/add_table.rb * /hbase/trunk/pom.xml Replace Jettison JSON encoding with Jackson in HLogPrettyPrinter Key: HBASE-4229 URL: https://issues.apache.org/jira/browse/HBASE-4229 Project: HBase Issue Type: Improvement Components: wal Reporter: Riley Patterson Assignee: Riley Patterson Priority: Trivial Fix For: 0.92.0 Attachments: HBASE-4229.patch HBase makes use of both jackson (in the region server) and jettison (in HLogPrettyPrinter) for JSON encoding. Jackson seems to be better maintained, so this patch standardizes by using jackson in HLogPrettyPrinter instead of jettison. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4220) Lots of DNS queries from client
[ https://issues.apache.org/jira/browse/HBASE-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088534#comment-13088534 ] ramkrishna.s.vasudevan commented on HBASE-4220: --- +1 Lots of DNS queries from client --- Key: HBASE-4220 URL: https://issues.apache.org/jira/browse/HBASE-4220 Project: HBase Issue Type: Bug Components: client Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 4220.txt In running a YCSB workload, I managed to DDOS a DNS server since it seems to be flooding lots of DNS requests. Installing nscd on the client machines improved throughput by a factor of 6 and stopped killing the server. These are long-running clients, so it's not clear why we do so many lookups. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4230) Compaction threads need names
[ https://issues.apache.org/jira/browse/HBASE-4230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088593#comment-13088593 ] Hudson commented on HBASE-4230: --- Integrated in HBase-TRUNK #2127 (See [https://builds.apache.org/job/HBase-TRUNK/2127/]) HBASE-4230 Compaction threads need names apurtell : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * /hbase/trunk/CHANGES.txt Compaction threads need names - Key: HBASE-4230 URL: https://issues.apache.org/jira/browse/HBASE-4230 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Andrew Purtell Fix For: 0.92.0 Attachments: HBASE-4230.patch The CompactSplitThread creates executors for doing compaction work, but threads end up named things like pool-2-thread-1 which isn't very useful. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subbu M Iyer updated HBASE-4213: Attachment: HBASE_Instant_schema_change-version_3_.patch Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4225) NoSuchColumnFamilyException in multi doesn't say which family is bad
[ https://issues.apache.org/jira/browse/HBASE-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reassigned HBASE-4225: - Assignee: ramkrishna.s.vasudevan NoSuchColumnFamilyException in multi doesn't say which family is bad Key: HBASE-4225 URL: https://issues.apache.org/jira/browse/HBASE-4225 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.90.5 It's kind of a dumb one, in HRegion.doMiniBatchPut we do: {code} LOG.warn(No such column family in batch put, nscf); batchOp.retCodes[lastIndexExclusive] = OperationStatusCode.BAD_FAMILY; {code} So we lose the family here, all we know is there's a bad one, that's what's in HRS.multi: {code} } else if (code == OperationStatusCode.BAD_FAMILY) { result = new NoSuchColumnFamilyException(); {code} We can't just throw the exception like that, we need to say which one is bad even if it requires testing all passed MultiActions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088462#comment-13088462 ] Subbu M Iyer commented on HBASE-4213: - Latest patch includes the following changes: 1. Load balancer will be turned off (if already enabled) before an instant schema change operation. 2. Load balancer will be turned on (if was on before the instant schema change). 3. Added a ZK based check instead of sleep timer for couple of tests. 4. Master stores the number of online RS while creating the schema change node. On completion the master compares against this count (rather than current number of active RS which might have changed during the schema change operation) with number of active RS to determine if all the RS have completed the schema change operation. Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088733#comment-13088733 ] Ted Yu commented on HBASE-4213: --- bq. 4. Master stores the number of online RS while creating the schema change node. Suppose RS1 was included in this count. Later it crashed and was brought up (either manually or by automated script). We should be able to tell that RS1 has completed schema change (by loading updated HTD). My suggestion is to record the names of region servers in the above step. Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4215) RS requestsPerSecond counter seems to be off
[ https://issues.apache.org/jira/browse/HBASE-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reassigned HBASE-4215: - Assignee: subramanian raghunathan (was: ramkrishna.s.vasudevan) RS requestsPerSecond counter seems to be off Key: HBASE-4215 URL: https://issues.apache.org/jira/browse/HBASE-4215 Project: HBase Issue Type: Bug Components: metrics Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: subramanian raghunathan Priority: Critical Fix For: 0.92.0 In testing trunk, I had YCSB reporting some 40,000 requests/second, but the summary info on the master webpage was consistently indicating somewhere around 3x that. I'm guessing that we may have a bug where we forgot to divide by time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4215) RS requestsPerSecond counter seems to be off
[ https://issues.apache.org/jira/browse/HBASE-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] subramanian raghunathan updated HBASE-4215: --- Attachment: HBASE-4215_trunk.patch RS requestsPerSecond counter seems to be off Key: HBASE-4215 URL: https://issues.apache.org/jira/browse/HBASE-4215 Project: HBase Issue Type: Bug Components: metrics Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: subramanian raghunathan Priority: Critical Fix For: 0.92.0 Attachments: HBASE-4215_trunk.patch In testing trunk, I had YCSB reporting some 40,000 requests/second, but the summary info on the master webpage was consistently indicating somewhere around 3x that. I'm guessing that we may have a bug where we forgot to divide by time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4215) RS requestsPerSecond counter seems to be off
[ https://issues.apache.org/jira/browse/HBASE-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] subramanian raghunathan updated HBASE-4215: --- Status: Patch Available (was: Open) RS requestsPerSecond counter seems to be off Key: HBASE-4215 URL: https://issues.apache.org/jira/browse/HBASE-4215 Project: HBase Issue Type: Bug Components: metrics Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: subramanian raghunathan Priority: Critical Fix For: 0.92.0 Attachments: HBASE-4215_trunk.patch In testing trunk, I had YCSB reporting some 40,000 requests/second, but the summary info on the master webpage was consistently indicating somewhere around 3x that. I'm guessing that we may have a bug where we forgot to divide by time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4225) NoSuchColumnFamilyException in multi doesn't say which family is bad
[ https://issues.apache.org/jira/browse/HBASE-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-4225: -- Attachment: HBASE-4225_0.90.patch Submitting the patch for 0.90.x version as given in the defect description. NoSuchColumnFamilyException in multi doesn't say which family is bad Key: HBASE-4225 URL: https://issues.apache.org/jira/browse/HBASE-4225 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.90.5 Attachments: HBASE-4225_0.90.patch It's kind of a dumb one, in HRegion.doMiniBatchPut we do: {code} LOG.warn(No such column family in batch put, nscf); batchOp.retCodes[lastIndexExclusive] = OperationStatusCode.BAD_FAMILY; {code} So we lose the family here, all we know is there's a bad one, that's what's in HRS.multi: {code} } else if (code == OperationStatusCode.BAD_FAMILY) { result = new NoSuchColumnFamilyException(); {code} We can't just throw the exception like that, we need to say which one is bad even if it requires testing all passed MultiActions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4225) NoSuchColumnFamilyException in multi doesn't say which family is bad
[ https://issues.apache.org/jira/browse/HBASE-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-4225: -- Status: Patch Available (was: Open) NoSuchColumnFamilyException in multi doesn't say which family is bad Key: HBASE-4225 URL: https://issues.apache.org/jira/browse/HBASE-4225 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.90.5 Attachments: HBASE-4225_0.90.patch It's kind of a dumb one, in HRegion.doMiniBatchPut we do: {code} LOG.warn(No such column family in batch put, nscf); batchOp.retCodes[lastIndexExclusive] = OperationStatusCode.BAD_FAMILY; {code} So we lose the family here, all we know is there's a bad one, that's what's in HRS.multi: {code} } else if (code == OperationStatusCode.BAD_FAMILY) { result = new NoSuchColumnFamilyException(); {code} We can't just throw the exception like that, we need to say which one is bad even if it requires testing all passed MultiActions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4225) NoSuchColumnFamilyException in multi doesn't say which family is bad
[ https://issues.apache.org/jira/browse/HBASE-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088804#comment-13088804 ] Ted Yu commented on HBASE-4225: --- For put(PairPut, Integer[] putsAndLocks): {code} + * @return Returns a Pair object containing the operation status code along with the exception {code} The return value is actually an array of Pairs. Good work. NoSuchColumnFamilyException in multi doesn't say which family is bad Key: HBASE-4225 URL: https://issues.apache.org/jira/browse/HBASE-4225 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.90.5 Attachments: HBASE-4225_0.90.patch It's kind of a dumb one, in HRegion.doMiniBatchPut we do: {code} LOG.warn(No such column family in batch put, nscf); batchOp.retCodes[lastIndexExclusive] = OperationStatusCode.BAD_FAMILY; {code} So we lose the family here, all we know is there's a bad one, that's what's in HRS.multi: {code} } else if (code == OperationStatusCode.BAD_FAMILY) { result = new NoSuchColumnFamilyException(); {code} We can't just throw the exception like that, we need to say which one is bad even if it requires testing all passed MultiActions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4225) NoSuchColumnFamilyException in multi doesn't say which family is bad
[ https://issues.apache.org/jira/browse/HBASE-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-4225: -- Attachment: HBASE-4225_0.90_1.patch @Ted, Changed the javadoc as per the comment.:) NoSuchColumnFamilyException in multi doesn't say which family is bad Key: HBASE-4225 URL: https://issues.apache.org/jira/browse/HBASE-4225 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.90.5 Attachments: HBASE-4225_0.90.patch, HBASE-4225_0.90_1.patch It's kind of a dumb one, in HRegion.doMiniBatchPut we do: {code} LOG.warn(No such column family in batch put, nscf); batchOp.retCodes[lastIndexExclusive] = OperationStatusCode.BAD_FAMILY; {code} So we lose the family here, all we know is there's a bad one, that's what's in HRS.multi: {code} } else if (code == OperationStatusCode.BAD_FAMILY) { result = new NoSuchColumnFamilyException(); {code} We can't just throw the exception like that, we need to say which one is bad even if it requires testing all passed MultiActions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4225) NoSuchColumnFamilyException in multi doesn't say which family is bad
[ https://issues.apache.org/jira/browse/HBASE-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-4225: -- Status: Open (was: Patch Available) NoSuchColumnFamilyException in multi doesn't say which family is bad Key: HBASE-4225 URL: https://issues.apache.org/jira/browse/HBASE-4225 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.90.5 Attachments: HBASE-4225_0.90.patch, HBASE-4225_0.90_1.patch It's kind of a dumb one, in HRegion.doMiniBatchPut we do: {code} LOG.warn(No such column family in batch put, nscf); batchOp.retCodes[lastIndexExclusive] = OperationStatusCode.BAD_FAMILY; {code} So we lose the family here, all we know is there's a bad one, that's what's in HRS.multi: {code} } else if (code == OperationStatusCode.BAD_FAMILY) { result = new NoSuchColumnFamilyException(); {code} We can't just throw the exception like that, we need to say which one is bad even if it requires testing all passed MultiActions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Delta Encoding of KeyValues (aka prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088827#comment-13088827 ] Jacek Migdal commented on HBASE-4218: - Regarding variable byte encoding. There is also another option than VInt and FInt: within a block have the same width of int, but it could be different across blocks. * exploit similarity of data within given block * usually have the same size as VInt * few branches * the key value format is not uniform across all of the data Having said that, in many Key Values there are only a few different sizes. That allows even more efficient encoding. On the other hand, when value lengths are getting longer, they vary a lot. But in that case keys are a tiny percent of whole file, so any savings from VB will be insignificant. Your mileage may vary. Delta Encoding of KeyValues (aka prefix compression) - Key: HBASE-4218 URL: https://issues.apache.org/jira/browse/HBASE-4218 Project: HBase Issue Type: Improvement Components: io Reporter: Jacek Migdal Labels: compression A compression for keys. Keys are sorted in HFile and they are usually very similar. Because of that, it is possible to design better compression than general purpose algorithms, It is an additional step designed to be used in memory. It aims to save memory in cache as well as speeding seeks within HFileBlocks. It should improve performance a lot, if key lengths are larger than value lengths. For example, it makes a lot of sense to use it when value is a counter. Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) shows that I could achieve decent level of compression: key compression ratio: 92% total compression ratio: 85% LZO on the same data: 85% LZO after delta encoding: 91% While having much better performance (20-80% faster decompression ratio than LZO). Moreover, it should allow far more efficient seeking which should improve performance a bit. It seems that a simple compression algorithms are good enough. Most of the savings are due to prefix compression, int128 encoding, timestamp diffs and bitfields to avoid duplication. That way, comparisons of compressed data can be much faster than a byte comparator (thanks to prefix compression and bitfields). In order to implement it in HBase two important changes in design will be needed: -solidify interface to HFileBlock / HFileReader Scanner to provide seeking and iterating; access to uncompressed buffer in HFileBlock will have bad performance -extend comparators to support comparison assuming that N first bytes are equal (or some fields are equal) Link to a discussion about something similar: http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windowssubj=Re+prefix+compression -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4065) TableOutputFormat ignores failure to create table instance
[ https://issues.apache.org/jira/browse/HBASE-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088864#comment-13088864 ] Ted Yu commented on HBASE-4065: --- Integrated to branch and TRUNK. Thanks for the patch Brock. TableOutputFormat ignores failure to create table instance -- Key: HBASE-4065 URL: https://issues.apache.org/jira/browse/HBASE-4065 Project: HBase Issue Type: Bug Affects Versions: 0.90.3 Reporter: Todd Lipcon Assignee: Brock Noland Fix For: 0.94.0 Attachments: HBASE-4065.1.patch, HBASE-4065.2.patch If TableOutputFormat in the new API fails to create a table, it simply logs this at ERROR level and then continues on its way. Then, the first write() to the table will throw a NPE since table hasn't been set. Instead, it should probably rethrow the exception as a RuntimeException in setConf, or do what the old-API TOF does and not create the HTable instance until getRecordWriter, where it can throw an IOE. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4236) Don't lock the stream while serializing the response
[ https://issues.apache.org/jira/browse/HBASE-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4236: -- Attachment: 4236.txt Don't lock the stream while serializing the response Key: HBASE-4236 URL: https://issues.apache.org/jira/browse/HBASE-4236 Project: HBase Issue Type: Improvement Components: ipc Affects Versions: 0.90.4 Reporter: Benoit Sigoure Assignee: Benoit Sigoure Priority: Minor Attachments: 4236.txt It is not necessary to hold the lock on the stream while the response is being serialized. This unnecessarily prevents serializing responses in parallel. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3842) Refactor Coprocessor Compaction API
[ https://issues.apache.org/jira/browse/HBASE-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-3842: - Priority: Critical (was: Major) We have a need for this internally, so I'll be working up a patch this week. As part of the coprocessor API, I think it's critical to get this in 0.92. Refactor Coprocessor Compaction API --- Key: HBASE-3842 URL: https://issues.apache.org/jira/browse/HBASE-3842 Project: HBase Issue Type: Improvement Components: coprocessors, regionserver Affects Versions: 0.92.0 Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Priority: Critical Labels: compaction Fix For: 0.92.0 After HBASE-3797, the compaction logic flow has been significantly altered. Because of this, the current compaction coprocessor API is insufficient for gaining full insight into compaction requests/results. Refactor coprocessor API after HBASE-3797 is committed to be more extensible and increase visibility. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4236) Don't lock the stream while serializing the response
[ https://issues.apache.org/jira/browse/HBASE-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088919#comment-13088919 ] Ted Yu commented on HBASE-4236: --- Committed to TRUNK. Thanks for the patch Benoit. Don't lock the stream while serializing the response Key: HBASE-4236 URL: https://issues.apache.org/jira/browse/HBASE-4236 Project: HBase Issue Type: Improvement Components: ipc Affects Versions: 0.90.4 Reporter: Benoit Sigoure Assignee: Benoit Sigoure Priority: Minor Attachments: 4236.txt It is not necessary to hold the lock on the stream while the response is being serialized. This unnecessarily prevents serializing responses in parallel. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4237) Directly remove the call being handled from the map of outstanding RPCs
[ https://issues.apache.org/jira/browse/HBASE-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089005#comment-13089005 ] Ted Yu commented on HBASE-4237: --- Integrated to TRUNK. Thanks for the patch Benoit. Directly remove the call being handled from the map of outstanding RPCs --- Key: HBASE-4237 URL: https://issues.apache.org/jira/browse/HBASE-4237 Project: HBase Issue Type: Improvement Components: ipc Affects Versions: 0.90.4 Reporter: Benoit Sigoure Assignee: Benoit Sigoure Priority: Minor The client has to maintain a map of RPC ID to `Call' object for this RPC, for every outstanding RPC. When receiving a response, the client was getting the `Call' out of the map (one O(log n) operation) and then removing it from the map (another O(log n) operation). There is no benefit in not removing it directly from the map. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4167) Potential leak of HTable instances when using HTablePool with PoolType.ThreadLocal
[ https://issues.apache.org/jira/browse/HBASE-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-4167: - Assignee: Karthick Sankarachary Potential leak of HTable instances when using HTablePool with PoolType.ThreadLocal -- Key: HBASE-4167 URL: https://issues.apache.org/jira/browse/HBASE-4167 Project: HBase Issue Type: Bug Components: client Reporter: Gary Helmling Assignee: Karthick Sankarachary Fix For: 0.92.0 Attachments: HBASE-4167.patch (Initially discussed in HBASE-4150) In HTablePool, when obtaining a table: {code} private HTableInterface findOrCreateTable(String tableName) { HTableInterface table = tables.get(tableName); if (table == null) { table = createHTable(tableName); } return table; } {code} In the case of {{ThreadLocalPool}}, it seems like there's an exposure here between when the table is created initially and when {{ThreadLocalPool.put()}} is called to set the thread local variable (on {{PooledHTable.close()}}). Potential solution described by Karthick Sankarachary: For one thing, we might want to clear the tables variable when the {{HTablePool}} is closed (as shown below). For another, we should override ThreadLocalPool#get method so that it removes the resource, otherwise it might end up referencing a HTableInterface that's has been released. {code} 1 diff --git a/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java b/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java 2 index 952a3aa..c198f15 100755 3 --- a/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java 4 +++ b/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java 13 @@ -309,6 +310,7 @@ public class HTablePool implements Closeable { 14 for (String tableName : tables.keySet()) { 15closeTablePool(tableName); 16 } 17 +this.tables.clear(); 18} {code} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4167) Potential leak of HTable instances when using HTablePool with PoolType.ThreadLocal
[ https://issues.apache.org/jira/browse/HBASE-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089010#comment-13089010 ] Ted Yu commented on HBASE-4167: --- Integrated to TRUNK. Thanks for the patch Karthick. Potential leak of HTable instances when using HTablePool with PoolType.ThreadLocal -- Key: HBASE-4167 URL: https://issues.apache.org/jira/browse/HBASE-4167 Project: HBase Issue Type: Bug Components: client Reporter: Gary Helmling Assignee: Karthick Sankarachary Fix For: 0.92.0 Attachments: HBASE-4167.patch (Initially discussed in HBASE-4150) In HTablePool, when obtaining a table: {code} private HTableInterface findOrCreateTable(String tableName) { HTableInterface table = tables.get(tableName); if (table == null) { table = createHTable(tableName); } return table; } {code} In the case of {{ThreadLocalPool}}, it seems like there's an exposure here between when the table is created initially and when {{ThreadLocalPool.put()}} is called to set the thread local variable (on {{PooledHTable.close()}}). Potential solution described by Karthick Sankarachary: For one thing, we might want to clear the tables variable when the {{HTablePool}} is closed (as shown below). For another, we should override ThreadLocalPool#get method so that it removes the resource, otherwise it might end up referencing a HTableInterface that's has been released. {code} 1 diff --git a/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java b/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java 2 index 952a3aa..c198f15 100755 3 --- a/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java 4 +++ b/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java 13 @@ -309,6 +310,7 @@ public class HTablePool implements Closeable { 14 for (String tableName : tables.keySet()) { 15closeTablePool(tableName); 16 } 17 +this.tables.clear(); 18} {code} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4236) Don't lock the stream while serializing the response
[ https://issues.apache.org/jira/browse/HBASE-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-4236. --- Resolution: Fixed Hadoop Flags: [Reviewed] Don't lock the stream while serializing the response Key: HBASE-4236 URL: https://issues.apache.org/jira/browse/HBASE-4236 Project: HBase Issue Type: Improvement Components: ipc Affects Versions: 0.90.4 Reporter: Benoit Sigoure Assignee: Benoit Sigoure Priority: Minor Attachments: 4236.txt It is not necessary to hold the lock on the stream while the response is being serialized. This unnecessarily prevents serializing responses in parallel. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-24) Scaling: Too many open file handles to datanodes
[ https://issues.apache.org/jira/browse/HBASE-24?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089042#comment-13089042 ] Alex Newman commented on HBASE-24: -- +conf); // defer opening streams is in the patch. Are we actually still defering. That seems wrong. Scaling: Too many open file handles to datanodes Key: HBASE-24 URL: https://issues.apache.org/jira/browse/HBASE-24 Project: HBase Issue Type: Bug Components: regionserver Reporter: stack Priority: Blocker Attachments: HBASE-823.patch, MonitoredReader.java We've been here before (HADOOP-2341). Today the rapleaf gave me an lsof listing from a regionserver. Had thousands of open sockets to datanodes all in ESTABLISHED and CLOSE_WAIT state. On average they seem to have about ten file descriptors/sockets open per region (They have 3 column families IIRC. Per family, can have between 1-5 or so mapfiles open per family -- 3 is max... but compacting we open a new one, etc.). They have thousands of regions. 400 regions -- ~100G, which is not that much -- takes about 4k open file handles. If they want a regionserver to server a decent disk worths -- 300-400G -- then thats maybe 1600 regions... 16k file handles. If more than just 3 column families. then we are in danger of blowing out limits if they are 32k. We've been here before with HADOOP-2341. A dfsclient that used non-blocking i/o would help applications like hbase (The datanode doesn't have this problem as bad -- CLOSE_WAIT on regionserver side, the bulk of the open fds in the rapleaf log, don't have a corresponding open resource on datanode end). Could also just open mapfiles as needed, but that'd kill our random read performance and its bad enough already. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4236) Don't lock the stream while serializing the response
[ https://issues.apache.org/jira/browse/HBASE-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089056#comment-13089056 ] Hudson commented on HBASE-4236: --- Integrated in HBase-TRUNK #2129 (See [https://builds.apache.org/job/HBase-TRUNK/2129/]) HBASE-4236 Don't lock the stream while serializing the response (Benoit Sigoure) tedyu : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java Don't lock the stream while serializing the response Key: HBASE-4236 URL: https://issues.apache.org/jira/browse/HBASE-4236 Project: HBase Issue Type: Improvement Components: ipc Affects Versions: 0.90.4 Reporter: Benoit Sigoure Assignee: Benoit Sigoure Priority: Minor Attachments: 4236.txt It is not necessary to hold the lock on the stream while the response is being serialized. This unnecessarily prevents serializing responses in parallel. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-3333) hbck -fix clearInMaster is commented out
[ https://issues.apache.org/jira/browse/HBASE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh resolved HBASE-. --- Resolution: Duplicate Assignee: Jonathan Hsieh hbck -fix clearInMaster is commented out Key: HBASE- URL: https://issues.apache.org/jira/browse/HBASE- Project: HBase Issue Type: Bug Affects Versions: 0.90.0 Reporter: Todd Lipcon Assignee: Jonathan Hsieh Priority: Critical In 0.90 rc the actual guts of the clearInMaster function are commented out, but it logs that it's doing something. We should either make the function do something, or log something like: hbck currently unable to fix region stuck in transition, rather than indicate we're doing something when in fact it's a noop. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089059#comment-13089059 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ --- (Updated 2011-08-22 21:55:23.810893) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes --- CacheableSerializer interface added. HFileBlock now fully serialized. Summary --- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) - conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 Diff: https://reviews.apache.org/r/1214/diff Testing --- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff Java offers the creation of direct byte buffers which are allocated outside of the heap. They need to be manually free'd, which can be accomplished using an documented {{clean}} method. The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3333) hbck -fix clearInMaster is commented out
[ https://issues.apache.org/jira/browse/HBASE-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089058#comment-13089058 ] Jonathan Hsieh commented on HBASE-: --- This was removed by HBASE-3337, and is in both the 0.90 branch and trunk. hbck -fix clearInMaster is commented out Key: HBASE- URL: https://issues.apache.org/jira/browse/HBASE- Project: HBase Issue Type: Bug Affects Versions: 0.90.0 Reporter: Todd Lipcon Priority: Critical In 0.90 rc the actual guts of the clearInMaster function are commented out, but it logs that it's doing something. We should either make the function do something, or log something like: hbck currently unable to fix region stuck in transition, rather than indicate we're doing something when in fact it's a noop. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Pi updated HBASE-4027: - Attachment: hbase-4027v13.1.diff latest version. new cacheableserialization interface. Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff Java offers the creation of direct byte buffers which are allocated outside of the heap. They need to be manually free'd, which can be accomplished using an documented {{clean}} method. The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4199) blockCache summary - backend
[ https://issues.apache.org/jira/browse/HBASE-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4199: -- Attachment: 4199.v5 Patch version 5 changes parameter arg to in/out for readFields()/write() blockCache summary - backend Key: HBASE-4199 URL: https://issues.apache.org/jira/browse/HBASE-4199 Project: HBase Issue Type: Sub-task Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: 4199.v5, java_HBASE_4199.patch, java_HBASE_4199_v2.patch, java_HBASE_4199_v3.patch, java_HBASE_4199_v4.patch This is the backend work for the blockCache summary. Change to BlockCache interface, Summarization in LruBlockCache, BlockCacheSummaryEntry, addition to HRegionInterface, and HRegionServer. This will NOT include any of the web UI or anything else like that. That is for another sub-task. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4071) Data GC: Remove all versions TTL EXCEPT the last written version
[ https://issues.apache.org/jira/browse/HBASE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089087#comment-13089087 ] jirapos...@reviews.apache.org commented on HBASE-4071: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1582/#review1591 --- Nice work! Some formatting things (didn't comment on some lines 80 chars) and need to update some of the javadocs for the new arguments. Will look through the patch one more time before I ask questions. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java https://reviews.apache.org/r/1582/#comment3577 line 80 chars http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java https://reviews.apache.org/r/1582/#comment3578 javadoc http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java https://reviews.apache.org/r/1582/#comment3582 seems like this change actually changes what the functionality / lines are between ColumnTracker vs. QueryMatcher. should you update the javadoc here to describe what this is not responsible for? http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java https://reviews.apache.org/r/1582/#comment3579 update the javadoc here... maybe explain why we pass ttl? seems odd that this removed timestamp from the API http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java https://reviews.apache.org/r/1582/#comment3581 update javadoc http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java https://reviews.apache.org/r/1582/#comment3580 long line http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java https://reviews.apache.org/r/1582/#comment3583 spaces between arguments: (minVersions, maxVersions, ttl) http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java https://reviews.apache.org/r/1582/#comment3584 update javadoc http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java https://reviews.apache.org/r/1582/#comment3585 i'm a little confused about this. this doesn't need to look at the number of included versions? you are only ever done if minVersions is explicitly set to 0 (disabled)? http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java https://reviews.apache.org/r/1582/#comment3587 this is a nice comment, but is this true? can't i safely delete anything that is expired once i have hit my min for this file? we may not evict _enough_ but we can at least evict something? unfortunately we don't use the query matcher here, but should be easy to add it here (someone else has added it here as part of some other changes, but they aren't committed yet and won't be for some time) http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java https://reviews.apache.org/r/1582/#comment3588 nice explanation - Jonathan On 2011-08-21 00:32:00, Lars Hofhansl wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1582/ bq. --- bq. bq. (Updated 2011-08-21 00:32:00) bq. bq. bq. Review request for hbase, Todd Lipcon, Michael Stack, and Ian Varley. bq. bq. bq. Summary bq. --- bq. bq. Allow enforcing a minimum number of versions when TTL is enable for a store. bq. The GC logic for both versions and TTL is unified inside the ColumnTrackers. bq. bq. bq. This addresses bug HBASE-4071. bq. https://issues.apache.org/jira/browse/HBASE-4071 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 1159836 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java 1159836 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java 1159836 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1159836 bq.
[jira] [Created] (HBASE-4238) CatalogJanitor can clear a daughter that split before processing its parent
CatalogJanitor can clear a daughter that split before processing its parent --- Key: HBASE-4238 URL: https://issues.apache.org/jira/browse/HBASE-4238 Project: HBase Issue Type: Bug Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Priority: Critical Fix For: 0.90.5 I didn't dig a lot into this issue, but by splitting a table twice in a row I was able to trigger a situation where a daughter of the first split was deleted by the CatalogJanitor before it processed its parent. Will post log in a comment. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4238) CatalogJanitor can clear a daughter that split before processing its parent
[ https://issues.apache.org/jira/browse/HBASE-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089099#comment-13089099 ] Jean-Daniel Cryans commented on HBASE-4238: --- Log of the issue (daughter region being d06b3b8c85af53ccbe139df7f0877e51): {quote} 2011-08-22 15:06:50,227 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Deleted daughter reference TestTable,,1314050587746.7ada9de242de30a4e91f00a8dfcbf693., qualifier=splitA, from parent TestTable,,1314050584480.d06b3b8c85af53ccbe139df7f0877e51. 2011-08-22 15:06:50,229 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Deleted daughter reference TestTable,013824,1314050587746.66c3c9f067203cfb7c0a541e9207766d., qualifier=splitB, from parent TestTable,,1314050584480.d06b3b8c85af53ccbe139df7f0877e51. 2011-08-22 15:06:50,230 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Deleting region TestTable,,1314050584480.d06b3b8c85af53ccbe139df7f0877e51. because daughter splits no longer hold references 2011-08-22 15:06:50,230 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: DELETING region file:/tmp/hbase-jdcryans/hbase/TestTable/d06b3b8c85af53ccbe139df7f0877e51 2011-08-22 15:06:50,234 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Deleted region TestTable,,1314050584480.d06b3b8c85af53ccbe139df7f0877e51. from META 2011-08-22 15:06:50,235 WARN org.apache.hadoop.hbase.master.CatalogJanitor: Daughter regiondir does not exist: file:/tmp/hbase-jdcryans/hbase/TestTable/d06b3b8c85af53ccbe139df7f0877e51 2011-08-22 15:11:50,220 WARN org.apache.hadoop.hbase.master.CatalogJanitor: Daughter regiondir does not exist: file:/tmp/hbase-jdcryans/hbase/TestTable/d06b3b8c85af53ccbe139df7f0877e51 2011-08-22 15:16:50,224 WARN org.apache.hadoop.hbase.master.CatalogJanitor: Daughter regiondir does not exist: file:/tmp/hbase-jdcryans/hbase/TestTable/d06b3b8c85af53ccbe139df7f0877e51 ... {quote} Those last lines will happen forever, and .META. will be littered with that row. CatalogJanitor can clear a daughter that split before processing its parent --- Key: HBASE-4238 URL: https://issues.apache.org/jira/browse/HBASE-4238 Project: HBase Issue Type: Bug Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Priority: Critical Fix For: 0.90.5 I didn't dig a lot into this issue, but by splitting a table twice in a row I was able to trigger a situation where a daughter of the first split was deleted by the CatalogJanitor before it processed its parent. Will post log in a comment. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4071) Data GC: Remove all versions TTL EXCEPT the last written version
[ https://issues.apache.org/jira/browse/HBASE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089103#comment-13089103 ] jirapos...@reviews.apache.org commented on HBASE-4071: -- bq. On 2011-08-22 22:24:47, Jonathan Gray wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java, line 205 bq. https://reviews.apache.org/r/1582/diff/4/?file=34041#file34041line205 bq. bq. i'm a little confused about this. this doesn't need to look at the number of included versions? you are only ever done if minVersions is explicitly set to 0 (disabled)? Yep... This is just to keep the current optimization where if we do not have minVersions to worry about (the default) we can bail early if the KV is expired. bq. On 2011-08-22 22:24:47, Jonathan Gray wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java, lines 24-41 bq. https://reviews.apache.org/r/1582/diff/4/?file=34037#file34037line24 bq. bq. seems like this change actually changes what the functionality / lines are between ColumnTracker vs. QueryMatcher. should you update the javadoc here to describe what this is not responsible for? Will fix this and all other Javadocs. bq. On 2011-08-22 22:24:47, Jonathan Gray wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java, line 499 bq. https://reviews.apache.org/r/1582/diff/4/?file=34042#file34042line499 bq. bq. this is a nice comment, but is this true? bq. bq. can't i safely delete anything that is expired once i have hit my min for this file? we may not evict _enough_ but we can at least evict something? bq. bq. unfortunately we don't use the query matcher here, but should be easy to add it here (someone else has added it here as part of some other changes, but they aren't committed yet and won't be for some time) You are right this can be optimized better as we discussed. Stack actually had the same comment on the issue. - Lars --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1582/#review1591 --- On 2011-08-21 00:32:00, Lars Hofhansl wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1582/ bq. --- bq. bq. (Updated 2011-08-21 00:32:00) bq. bq. bq. Review request for hbase, Todd Lipcon, Michael Stack, and Ian Varley. bq. bq. bq. Summary bq. --- bq. bq. Allow enforcing a minimum number of versions when TTL is enable for a store. bq. The GC logic for both versions and TTL is unified inside the ColumnTrackers. bq. bq. bq. This addresses bug HBASE-4071. bq. https://issues.apache.org/jira/browse/HBASE-4071 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 1159836 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java 1159836 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java 1159836 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1159836 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1159836 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java 1159836 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1159836 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1159836 bq.http://svn.apache.org/repos/asf/hbase/trunk/src/main/ruby/hbase/admin.rb 1159836 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 1159836 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java 1159836 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java 1159836 bq. bq. Diff:
[jira] [Commented] (HBASE-4199) blockCache summary - backend
[ https://issues.apache.org/jira/browse/HBASE-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089104#comment-13089104 ] Jean-Daniel Cryans commented on HBASE-4199: --- Still +1, good work! blockCache summary - backend Key: HBASE-4199 URL: https://issues.apache.org/jira/browse/HBASE-4199 Project: HBase Issue Type: Sub-task Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: 4199.v5, java_HBASE_4199.patch, java_HBASE_4199_v2.patch, java_HBASE_4199_v3.patch, java_HBASE_4199_v4.patch This is the backend work for the blockCache summary. Change to BlockCache interface, Summarization in LruBlockCache, BlockCacheSummaryEntry, addition to HRegionInterface, and HRegionServer. This will NOT include any of the web UI or anything else like that. That is for another sub-task. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4199) blockCache summary - backend
[ https://issues.apache.org/jira/browse/HBASE-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089121#comment-13089121 ] Ted Yu commented on HBASE-4199: --- Integrated to TRUNK. Thanks for the patch Doug. Thanks for the review Stack and J-D. blockCache summary - backend Key: HBASE-4199 URL: https://issues.apache.org/jira/browse/HBASE-4199 Project: HBase Issue Type: Sub-task Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: 4199.v5, java_HBASE_4199.patch, java_HBASE_4199_v2.patch, java_HBASE_4199_v3.patch, java_HBASE_4199_v4.patch This is the backend work for the blockCache summary. Change to BlockCache interface, Summarization in LruBlockCache, BlockCacheSummaryEntry, addition to HRegionInterface, and HRegionServer. This will NOT include any of the web UI or anything else like that. That is for another sub-task. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4239) HBASE-4012 introduced duplicate variable Bytes.LONG_BYTES
HBASE-4012 introduced duplicate variable Bytes.LONG_BYTES - Key: HBASE-4239 URL: https://issues.apache.org/jira/browse/HBASE-4239 Project: HBase Issue Type: Improvement Affects Versions: 0.92.0 Reporter: Jonathan Gray Assignee: Ted Yu Priority: Trivial Fix For: 0.92.0 HBASE-4012 introduced Bytes.LONG_SIZE. This is a duplicate of Bytes.SIZEOF_LONG. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089134#comment-13089134 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ --- (Updated 2011-08-22 23:09:44.370020) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes --- fixed concurrency issue. now with readwritelocks. Summary --- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) - CHANGES.txt 0bb7c65 conf/hbase-env.sh 2d55d27 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheColumnFamilySummary.java 34513f1 src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 8f5600d src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java a817c37 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSUtils.java d6fa01b src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/TestBlockCacheColumnFamilySummary.java cc4abc6 src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java f8a854c Diff: https://reviews.apache.org/r/1214/diff Testing --- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff Java offers the creation of direct byte buffers which are allocated outside of the heap. They need to be manually free'd, which can be accomplished using an documented {{clean}} method. The feature will be optional. After implementing, we can benchmark for differences in speed and garbage collection observances. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4239) HBASE-4012 introduced duplicate variable Bytes.LONG_BYTES
[ https://issues.apache.org/jira/browse/HBASE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4239: -- Attachment: 4239.txt Removes reference to LONG_BYTES HBASE-4012 introduced duplicate variable Bytes.LONG_BYTES - Key: HBASE-4239 URL: https://issues.apache.org/jira/browse/HBASE-4239 Project: HBase Issue Type: Improvement Affects Versions: 0.92.0 Reporter: Jonathan Gray Assignee: Ted Yu Priority: Trivial Fix For: 0.92.0 Attachments: 4239.txt HBASE-4012 introduced Bytes.LONG_SIZE. This is a duplicate of Bytes.SIZEOF_LONG. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HBASE-4239) HBASE-4012 introduced duplicate variable Bytes.LONG_BYTES
[ https://issues.apache.org/jira/browse/HBASE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-4239 started by Ted Yu. HBASE-4012 introduced duplicate variable Bytes.LONG_BYTES - Key: HBASE-4239 URL: https://issues.apache.org/jira/browse/HBASE-4239 Project: HBase Issue Type: Improvement Affects Versions: 0.92.0 Reporter: Jonathan Gray Assignee: Ted Yu Priority: Trivial Fix For: 0.92.0 Attachments: 4239.txt HBASE-4012 introduced Bytes.LONG_SIZE. This is a duplicate of Bytes.SIZEOF_LONG. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4117) Slow Query Log
[ https://issues.apache.org/jira/browse/HBASE-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089146#comment-13089146 ] jirapos...@reviews.apache.org commented on HBASE-4117: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1621/ --- (Updated 2011-08-22 23:25:12.626245) Review request for hbase. Changes --- Addressed stack's comments from last week, cleaned up code, standardized many fields in the JSON output. Summary --- Exposes JSON-parseable fingerprint and details for queries that take longer than a configurable threshold time. The exposure is currently to the main regionserver log, along with a (operationTooSlow) tag which allows it to be grepped out and easily aggregated and/or monitored in administrator scripts. The patch also provides a standard way to extract fingerprint and detail information of interest by requiring each Operation to provide a fingerprint map and a details map, which will be a superset of the fingerprint. This addresses bug HBASE-4117. https://issues.apache.org/jira/browse/HBASE-4117 Diffs - /src/main/java/org/apache/hadoop/hbase/KeyValue.java 1160468 /src/main/java/org/apache/hadoop/hbase/client/Delete.java 1160468 /src/main/java/org/apache/hadoop/hbase/client/Get.java 1160468 /src/main/java/org/apache/hadoop/hbase/client/MultiAction.java 1160468 /src/main/java/org/apache/hadoop/hbase/client/MultiPut.java 1160468 /src/main/java/org/apache/hadoop/hbase/client/Operation.java PRE-CREATION /src/main/java/org/apache/hadoop/hbase/client/Put.java 1160468 /src/main/java/org/apache/hadoop/hbase/client/Scan.java 1160468 /src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1160468 /src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 1160468 /src/test/java/org/apache/hadoop/hbase/client/TestOperation.java PRE-CREATION Diff: https://reviews.apache.org/r/1621/diff Testing --- Monitored get and put latency with and without the patch using reasonable hbase.ipc.warn.response.time settings (1000ms, 2000ms). Performance was not noticeably impacted. Thanks, Riley Slow Query Log -- Key: HBASE-4117 URL: https://issues.apache.org/jira/browse/HBASE-4117 Project: HBase Issue Type: New Feature Components: ipc Reporter: Riley Patterson Assignee: Riley Patterson Priority: Minor Labels: client, ipc Attachments: HBASE-4117.patch Produce log messages for slow queries. The RPC server will decide what is slow based on a configurable warn response time parameter. Queries designated as slow will then output a response too slow message followed by a fingerprint of the query, and a summary limited in size by another configurable parameter (to limit log spamming). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4071) Data GC: Remove all versions TTL EXCEPT the last written version
[ https://issues.apache.org/jira/browse/HBASE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089145#comment-13089145 ] jirapos...@reviews.apache.org commented on HBASE-4071: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1582/ --- (Updated 2011-08-22 23:24:57.784160) Review request for hbase, Todd Lipcon, Michael Stack, and Ian Varley. Changes --- Fixed comments, Javadocs, and linebreaks. Summary --- Allow enforcing a minimum number of versions when TTL is enable for a store. The GC logic for both versions and TTL is unified inside the ColumnTrackers. This addresses bug HBASE-4071. https://issues.apache.org/jira/browse/HBASE-4071 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/ruby/hbase/admin.rb 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java 1160440 Diff: https://reviews.apache.org/r/1582/diff Testing --- Ran all tests. I get error (not failures) from two: TestDistributedLogSplitting and TestHTablePool. Both fail with or without my changes. New tests: TestMinVersions. Thanks, Lars Data GC: Remove all versions TTL EXCEPT the last written version -- Key: HBASE-4071 URL: https://issues.apache.org/jira/browse/HBASE-4071 Project: HBase Issue Type: New Feature Reporter: stack Attachments: MinVersions.diff We were chatting today about our backup cluster. What we want is to be able to restore the dataset from any point of time but only within a limited timeframe -- say one week. Thereafter, if the versions are older than one week, rather than as we do with TTL where we let go of all versions older than TTL, instead, let go of all versions EXCEPT the last one written. So, its like versions==1 when TTL one week. We want to allow that if an error is caught within a week of its happening -- user mistakenly removes a critical table -- then we'll be able to restore up the the moment just before catastrophe hit otherwise, we keep one version only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4239) HBASE-4012 introduced duplicate variable Bytes.LONG_BYTES
[ https://issues.apache.org/jira/browse/HBASE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4239: -- Status: Patch Available (was: In Progress) HBASE-4012 introduced duplicate variable Bytes.LONG_BYTES - Key: HBASE-4239 URL: https://issues.apache.org/jira/browse/HBASE-4239 Project: HBase Issue Type: Improvement Affects Versions: 0.92.0 Reporter: Jonathan Gray Assignee: Ted Yu Priority: Trivial Fix For: 0.92.0 Attachments: 4239.txt HBASE-4012 introduced Bytes.LONG_SIZE. This is a duplicate of Bytes.SIZEOF_LONG. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089148#comment-13089148 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1594 --- Looking good! Still some lines 80 chars and whitespace to fix, but otherwise not much to comment on. As we discussed, I find the Cacheable interface name a bit confusing. Maybe rename it, or just fill the classes and methods with awesome javadoc that explains how it all works :) Also, should probably add a section to the book about this? Some of these parameters are pretty important and the defaults could be wasteful for some of the default use cases (for example, always giving 20% of heap to blocks 1.1X block size) src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java https://reviews.apache.org/r/1214/#comment3591 is it required still that the block contents be wrapped in a ByteBuffer? does Cacheable enforce this? if not, should this javadoc be updated? src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java https://reviews.apache.org/r/1214/#comment3592 extra newline (maybe move it to below the public class). also missing a space after CacheStats src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment3594 looks like this javadoc is accidentally applied on the CacheStats private var instead of constructor src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java https://reviews.apache.org/r/1214/#comment3595 should this be moved somewhere else? seems strange to have all this code up in the top of the class in the middle of class variable members src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java https://reviews.apache.org/r/1214/#comment3596 is this necessary since we throw a runtime exception? could this leave the RS in a weird state? should we actually halt or abort? or will that happen when this gets thrown? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java https://reviews.apache.org/r/1214/#comment3599 i think this is the only place you use the term chunks. should we just call the slab chunks blocks? or should we call them chunks everywhere to avoid confusion with hfile blocks? although they are kinda the same :) src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java https://reviews.apache.org/r/1214/#comment3597 javadoc src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java https://reviews.apache.org/r/1214/#comment3600 add @param @return javadoc - Jonathan On 2011-08-22 23:09:44, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-22 23:09:44) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.CHANGES.txt 0bb7c65 bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheColumnFamilySummary.java 34513f1 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.
[jira] [Commented] (HBASE-4239) HBASE-4012 introduced duplicate variable Bytes.LONG_BYTES
[ https://issues.apache.org/jira/browse/HBASE-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089149#comment-13089149 ] Jonathan Gray commented on HBASE-4239: -- +1 HBASE-4012 introduced duplicate variable Bytes.LONG_BYTES - Key: HBASE-4239 URL: https://issues.apache.org/jira/browse/HBASE-4239 Project: HBase Issue Type: Improvement Affects Versions: 0.92.0 Reporter: Jonathan Gray Assignee: Ted Yu Priority: Trivial Fix For: 0.92.0 Attachments: 4239.txt HBASE-4012 introduced Bytes.LONG_SIZE. This is a duplicate of Bytes.SIZEOF_LONG. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4240) Allow Loadbalancer to be pluggable.
Allow Loadbalancer to be pluggable. --- Key: HBASE-4240 URL: https://issues.apache.org/jira/browse/HBASE-4240 Project: HBase Issue Type: New Feature Components: master Affects Versions: 0.94.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.94.0 Everyone seems to want something different from a load balancer. People want low latency, simplicity, and total control. It seems like at some point the load balancer can't be all things to all people. Something akin to what hadoop JT's pluggable scheduler seems like it will enable all solutions without making the code much more complex. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4240) Allow Loadbalancer to be pluggable.
[ https://issues.apache.org/jira/browse/HBASE-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-4240: - Attachment: HBASE-4240-0.patch Allow Loadbalancer to be pluggable. --- Key: HBASE-4240 URL: https://issues.apache.org/jira/browse/HBASE-4240 Project: HBase Issue Type: New Feature Components: master Affects Versions: 0.94.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.94.0 Attachments: HBASE-4240-0.patch Everyone seems to want something different from a load balancer. People want low latency, simplicity, and total control. It seems like at some point the load balancer can't be all things to all people. Something akin to what hadoop JT's pluggable scheduler seems like it will enable all solutions without making the code much more complex. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4240) Allow Loadbalancer to be pluggable.
[ https://issues.apache.org/jira/browse/HBASE-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089156#comment-13089156 ] Elliott Clark commented on HBASE-4240: -- Attached patch. It's not well polished right now. It needs some javadoc love and I need to add some other tests to show how to plug. Comments are welcomed. Allow Loadbalancer to be pluggable. --- Key: HBASE-4240 URL: https://issues.apache.org/jira/browse/HBASE-4240 Project: HBase Issue Type: New Feature Components: master Affects Versions: 0.94.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.94.0 Attachments: HBASE-4240-0.patch Everyone seems to want something different from a load balancer. People want low latency, simplicity, and total control. It seems like at some point the load balancer can't be all things to all people. Something akin to what hadoop JT's pluggable scheduler seems like it will enable all solutions without making the code much more complex. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4071) Data GC: Remove all versions TTL EXCEPT the last written version
[ https://issues.apache.org/jira/browse/HBASE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089160#comment-13089160 ] jirapos...@reviews.apache.org commented on HBASE-4071: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1582/#review1595 --- http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java https://reviews.apache.org/r/1582/#comment3601 these are flipped in order from the signature http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java https://reviews.apache.org/r/1582/#comment3602 @return javadoc doesn't include the variable name (on the @param does) - Jonathan On 2011-08-22 23:24:57, Lars Hofhansl wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1582/ bq. --- bq. bq. (Updated 2011-08-22 23:24:57) bq. bq. bq. Review request for hbase, Todd Lipcon, Michael Stack, and Ian Varley. bq. bq. bq. Summary bq. --- bq. bq. Allow enforcing a minimum number of versions when TTL is enable for a store. bq. The GC logic for both versions and TTL is unified inside the ColumnTrackers. bq. bq. bq. This addresses bug HBASE-4071. bq. https://issues.apache.org/jira/browse/HBASE-4071 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1160440 bq.http://svn.apache.org/repos/asf/hbase/trunk/src/main/ruby/hbase/admin.rb 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java 1160440 bq. bq. Diff: https://reviews.apache.org/r/1582/diff bq. bq. bq. Testing bq. --- bq. bq. Ran all tests. I get error (not failures) from two: TestDistributedLogSplitting and TestHTablePool. Both fail with or without my changes. bq. New tests: TestMinVersions. bq. bq. bq. Thanks, bq. bq. Lars bq. bq. Data GC: Remove all versions TTL EXCEPT the last written version -- Key: HBASE-4071 URL: https://issues.apache.org/jira/browse/HBASE-4071 Project: HBase Issue Type: New Feature Reporter: stack Attachments: MinVersions.diff We were chatting today about our backup cluster. What we want is to be able to restore the dataset from any point of time but only within a limited timeframe -- say one week. Thereafter, if the versions are older than one week, rather than as we do with TTL where we let go of all versions older than TTL, instead, let go of all versions EXCEPT the last one written. So, its like versions==1 when TTL one week. We want to allow that if an error is caught within a week of its happening -- user mistakenly removes a critical table -- then we'll be able to restore up the the moment just before catastrophe hit otherwise, we keep one version only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign)
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089188#comment-13089188 ] Subbu M Iyer commented on HBASE-4213: - Sure. We have two approaches. 1730 + master failover Or 4213 + throttling + progress status. Let me know which one we want and I can address it accordingly. One question I have is: Do we really need to go through bulk unassign/assign(through ZK) for schema changes ? What is the real drawback of simply signaling the RS cloud to refresh their regions on schema changes? Sent from my iPhone Support instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.92.0 Attachments: HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4237) Directly remove the call being handled from the map of outstanding RPCs
[ https://issues.apache.org/jira/browse/HBASE-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089194#comment-13089194 ] Hudson commented on HBASE-4237: --- Integrated in HBase-TRUNK #2130 (See [https://builds.apache.org/job/HBase-TRUNK/2130/]) HBASE-4237 Directly remove the call being handled from the map of outstanding RPCs (Benoit Sigoure) tedyu : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java Directly remove the call being handled from the map of outstanding RPCs --- Key: HBASE-4237 URL: https://issues.apache.org/jira/browse/HBASE-4237 Project: HBase Issue Type: Improvement Components: ipc Affects Versions: 0.90.4 Reporter: Benoit Sigoure Assignee: Benoit Sigoure Priority: Minor The client has to maintain a map of RPC ID to `Call' object for this RPC, for every outstanding RPC. When receiving a response, the client was getting the `Call' out of the map (one O(log n) operation) and then removing it from the map (another O(log n) operation). There is no benefit in not removing it directly from the map. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4167) Potential leak of HTable instances when using HTablePool with PoolType.ThreadLocal
[ https://issues.apache.org/jira/browse/HBASE-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089195#comment-13089195 ] Hudson commented on HBASE-4167: --- Integrated in HBase-TRUNK #2130 (See [https://builds.apache.org/job/HBase-TRUNK/2130/]) HBASE-4167 Potential leak of HTable instances when using HTablePool with PoolType.ThreadLocal (Karthick Sankarachary) tedyu : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java Potential leak of HTable instances when using HTablePool with PoolType.ThreadLocal -- Key: HBASE-4167 URL: https://issues.apache.org/jira/browse/HBASE-4167 Project: HBase Issue Type: Bug Components: client Reporter: Gary Helmling Assignee: Karthick Sankarachary Fix For: 0.92.0 Attachments: HBASE-4167.patch (Initially discussed in HBASE-4150) In HTablePool, when obtaining a table: {code} private HTableInterface findOrCreateTable(String tableName) { HTableInterface table = tables.get(tableName); if (table == null) { table = createHTable(tableName); } return table; } {code} In the case of {{ThreadLocalPool}}, it seems like there's an exposure here between when the table is created initially and when {{ThreadLocalPool.put()}} is called to set the thread local variable (on {{PooledHTable.close()}}). Potential solution described by Karthick Sankarachary: For one thing, we might want to clear the tables variable when the {{HTablePool}} is closed (as shown below). For another, we should override ThreadLocalPool#get method so that it removes the resource, otherwise it might end up referencing a HTableInterface that's has been released. {code} 1 diff --git a/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java b/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java 2 index 952a3aa..c198f15 100755 3 --- a/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java 4 +++ b/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java 13 @@ -309,6 +310,7 @@ public class HTablePool implements Closeable { 14 for (String tableName : tables.keySet()) { 15closeTablePool(tableName); 16 } 17 +this.tables.clear(); 18} {code} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4167) Potential leak of HTable instances when using HTablePool with PoolType.ThreadLocal
[ https://issues.apache.org/jira/browse/HBASE-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-4167. --- Resolution: Fixed Hadoop Flags: [Reviewed] Potential leak of HTable instances when using HTablePool with PoolType.ThreadLocal -- Key: HBASE-4167 URL: https://issues.apache.org/jira/browse/HBASE-4167 Project: HBase Issue Type: Bug Components: client Reporter: Gary Helmling Assignee: Karthick Sankarachary Fix For: 0.92.0 Attachments: HBASE-4167.patch (Initially discussed in HBASE-4150) In HTablePool, when obtaining a table: {code} private HTableInterface findOrCreateTable(String tableName) { HTableInterface table = tables.get(tableName); if (table == null) { table = createHTable(tableName); } return table; } {code} In the case of {{ThreadLocalPool}}, it seems like there's an exposure here between when the table is created initially and when {{ThreadLocalPool.put()}} is called to set the thread local variable (on {{PooledHTable.close()}}). Potential solution described by Karthick Sankarachary: For one thing, we might want to clear the tables variable when the {{HTablePool}} is closed (as shown below). For another, we should override ThreadLocalPool#get method so that it removes the resource, otherwise it might end up referencing a HTableInterface that's has been released. {code} 1 diff --git a/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java b/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java 2 index 952a3aa..c198f15 100755 3 --- a/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java 4 +++ b/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java 13 @@ -309,6 +310,7 @@ public class HTablePool implements Closeable { 14 for (String tableName : tables.keySet()) { 15closeTablePool(tableName); 16 } 17 +this.tables.clear(); 18} {code} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4237) Directly remove the call being handled from the map of outstanding RPCs
[ https://issues.apache.org/jira/browse/HBASE-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-4237. --- Resolution: Fixed Hadoop Flags: [Reviewed] Directly remove the call being handled from the map of outstanding RPCs --- Key: HBASE-4237 URL: https://issues.apache.org/jira/browse/HBASE-4237 Project: HBase Issue Type: Improvement Components: ipc Affects Versions: 0.90.4 Reporter: Benoit Sigoure Assignee: Benoit Sigoure Priority: Minor The client has to maintain a map of RPC ID to `Call' object for this RPC, for every outstanding RPC. When receiving a response, the client was getting the `Call' out of the map (one O(log n) operation) and then removing it from the map (another O(log n) operation). There is no benefit in not removing it directly from the map. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4124) ZK restarted while assigning a region, new active HM re-assign it but the RS warned 'already online on this server'.
[ https://issues.apache.org/jira/browse/HBASE-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gaojinchao reassigned HBASE-4124: - Assignee: gaojinchao ZK restarted while assigning a region, new active HM re-assign it but the RS warned 'already online on this server'. Key: HBASE-4124 URL: https://issues.apache.org/jira/browse/HBASE-4124 Project: HBase Issue Type: Bug Components: master Reporter: fulin wang Assignee: gaojinchao Fix For: 0.90.5 Attachments: HBASE-4124_Branch90V1_trial.patch, HBASE-4124_Branch90V2.patch, log.txt Original Estimate: 0.4h Remaining Estimate: 0.4h ZK restarted while assigning a region, new active HM re-assign it but the RS warned 'already online on this server'. Issue: The RS failed besause of 'already online on this server' and return; The HM can not receive the message and report 'Regions in transition timed out'. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4124) ZK restarted while assigning a region, new active HM re-assign it but the RS warned 'already online on this server'.
[ https://issues.apache.org/jira/browse/HBASE-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] gaojinchao updated HBASE-4124: -- Fix Version/s: 0.90.5 ZK restarted while assigning a region, new active HM re-assign it but the RS warned 'already online on this server'. Key: HBASE-4124 URL: https://issues.apache.org/jira/browse/HBASE-4124 Project: HBase Issue Type: Bug Components: master Reporter: fulin wang Fix For: 0.90.5 Attachments: HBASE-4124_Branch90V1_trial.patch, HBASE-4124_Branch90V2.patch, log.txt Original Estimate: 0.4h Remaining Estimate: 0.4h ZK restarted while assigning a region, new active HM re-assign it but the RS warned 'already online on this server'. Issue: The RS failed besause of 'already online on this server' and return; The HM can not receive the message and report 'Regions in transition timed out'. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4240) Allow Loadbalancer to be pluggable.
[ https://issues.apache.org/jira/browse/HBASE-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089207#comment-13089207 ] Ted Yu commented on HBASE-4240: --- TestLoadBalancer should be named TestDefaultLoadBalancer since factory isn't used to generate the balancer. A better name for hbase.maser.loadBalancer would be hbase.maser.loadbalancer.class Allow Loadbalancer to be pluggable. --- Key: HBASE-4240 URL: https://issues.apache.org/jira/browse/HBASE-4240 Project: HBase Issue Type: New Feature Components: master Affects Versions: 0.94.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.94.0 Attachments: HBASE-4240-0.patch Everyone seems to want something different from a load balancer. People want low latency, simplicity, and total control. It seems like at some point the load balancer can't be all things to all people. Something akin to what hadoop JT's pluggable scheduler seems like it will enable all solutions without making the code much more complex. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4240) Allow Loadbalancer to be pluggable.
[ https://issues.apache.org/jira/browse/HBASE-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-4240: - Attachment: HBASE-4240-1.patch Added more docs and changed the key. Allow Loadbalancer to be pluggable. --- Key: HBASE-4240 URL: https://issues.apache.org/jira/browse/HBASE-4240 Project: HBase Issue Type: New Feature Components: master Affects Versions: 0.94.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.94.0 Attachments: HBASE-4240-0.patch, HBASE-4240-1.patch Everyone seems to want something different from a load balancer. People want low latency, simplicity, and total control. It seems like at some point the load balancer can't be all things to all people. Something akin to what hadoop JT's pluggable scheduler seems like it will enable all solutions without making the code much more complex. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4012) Further optimize byte comparison methods
[ https://issues.apache.org/jira/browse/HBASE-4012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089228#comment-13089228 ] Hudson commented on HBASE-4012: --- Integrated in HBase-TRUNK #2131 (See [https://builds.apache.org/job/HBase-TRUNK/2131/]) HBASE-4239 HBASE-4012 introduced duplicate variable Bytes.LONG_BYTES tedyu : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/Bytes.java Further optimize byte comparison methods Key: HBASE-4012 URL: https://issues.apache.org/jira/browse/HBASE-4012 Project: HBase Issue Type: Improvement Components: util Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Ted Yu Priority: Minor Labels: noob Fix For: 0.92.0 Attachments: 4012-v2.txt, 4012.txt Guava uses some clever tricks with sun.misc.Unsafe to compare byte arrays about 100% faster than the naive byte-by-byte implementation: http://guava-libraries.googlecode.com/svn/trunk/guava/src/com/google/common/primitives/UnsignedBytes.java We should borrow this [Apache 2 licensed] code. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4240) Allow Loadbalancer to be pluggable.
[ https://issues.apache.org/jira/browse/HBASE-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089247#comment-13089247 ] Ted Yu commented on HBASE-4240: --- {code} + public ListHRegionInfo randomize(final ListHRegionInfo regions) { {code} The only reference to the above method I can find is in TestLoadBalancer. I think it is safe to remove it from interface and implementation. Allow Loadbalancer to be pluggable. --- Key: HBASE-4240 URL: https://issues.apache.org/jira/browse/HBASE-4240 Project: HBase Issue Type: New Feature Components: master Affects Versions: 0.94.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.94.0 Attachments: HBASE-4240-0.patch, HBASE-4240-1.patch Everyone seems to want something different from a load balancer. People want low latency, simplicity, and total control. It seems like at some point the load balancer can't be all things to all people. Something akin to what hadoop JT's pluggable scheduler seems like it will enable all solutions without making the code much more complex. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4240) Allow Loadbalancer to be pluggable.
[ https://issues.apache.org/jira/browse/HBASE-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089249#comment-13089249 ] Ted Yu commented on HBASE-4240: --- {code} + * Copyright 2010 The Apache Software Foundation {code} The year should be 2011. {code} + +public class TestDefaultLoadBalancer { {code} Please add javadoc for the above class. Allow Loadbalancer to be pluggable. --- Key: HBASE-4240 URL: https://issues.apache.org/jira/browse/HBASE-4240 Project: HBase Issue Type: New Feature Components: master Affects Versions: 0.94.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.94.0 Attachments: HBASE-4240-0.patch, HBASE-4240-1.patch Everyone seems to want something different from a load balancer. People want low latency, simplicity, and total control. It seems like at some point the load balancer can't be all things to all people. Something akin to what hadoop JT's pluggable scheduler seems like it will enable all solutions without making the code much more complex. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4241) Optimize flushing of the Store cache for max versions and (new) min versions
Optimize flushing of the Store cache for max versions and (new) min versions Key: HBASE-4241 URL: https://issues.apache.org/jira/browse/HBASE-4241 Project: HBase Issue Type: Sub-task Components: regionserver Affects Versions: 0.92.0 Reporter: Lars Hofhansl Priority: Minor As discussed with with Jon, there is room for improvement in how the memstore is flushed to disk. Currently only expired KVs are pruned before flushing, but we can also prune versions if we find at least maxVersions versions in the memstore. The same holds for the new minversion feature: If we find at least minVersion versions in the store we can remove all further versions that are expired. Generally we should use the same mechanism here that is used for Compaction. I.e. StoreScanner. We only need to add a scanner to Memstore that can scan along the current snapshot. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4240) Allow Loadbalancer to be pluggable.
[ https://issues.apache.org/jira/browse/HBASE-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-4240: - Attachment: HBASE-4240-2.patch Added Javadoc to TestDefaultLoadBalancer. Moved Copyright on changed files to 2011 Moved randomize to TestDefaultLoadBalancer. Allow Loadbalancer to be pluggable. --- Key: HBASE-4240 URL: https://issues.apache.org/jira/browse/HBASE-4240 Project: HBase Issue Type: New Feature Components: master Affects Versions: 0.94.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.94.0 Attachments: HBASE-4240-0.patch, HBASE-4240-1.patch, HBASE-4240-2.patch Everyone seems to want something different from a load balancer. People want low latency, simplicity, and total control. It seems like at some point the load balancer can't be all things to all people. Something akin to what hadoop JT's pluggable scheduler seems like it will enable all solutions without making the code much more complex. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4124) ZK restarted while assigning a region, new active HM re-assign it but the RS warned 'already online on this server'.
[ https://issues.apache.org/jira/browse/HBASE-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089261#comment-13089261 ] ramkrishna.s.vasudevan commented on HBASE-4124: --- @Gao {bq} step 3: startup master again . As per the scenario you have described when the master restarted the RS has it opened the region? I think the scenario here is RS is also dead. If so the assignment manager will try assigning it to a new RS. Do you think any problem here? If the RS is alive then the znode status will be OPENED state and the processRIT will take care of clearing the node as it is already opened. Could be be more clear on the state of RS after you killed the master and also on the state of znode in zookeeper for that region. ZK restarted while assigning a region, new active HM re-assign it but the RS warned 'already online on this server'. Key: HBASE-4124 URL: https://issues.apache.org/jira/browse/HBASE-4124 Project: HBase Issue Type: Bug Components: master Reporter: fulin wang Assignee: gaojinchao Fix For: 0.90.5 Attachments: HBASE-4124_Branch90V1_trial.patch, HBASE-4124_Branch90V2.patch, log.txt Original Estimate: 0.4h Remaining Estimate: 0.4h ZK restarted while assigning a region, new active HM re-assign it but the RS warned 'already online on this server'. Issue: The RS failed besause of 'already online on this server' and return; The HM can not receive the message and report 'Regions in transition timed out'. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4240) Allow Loadbalancer to be pluggable.
[ https://issues.apache.org/jira/browse/HBASE-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089265#comment-13089265 ] Ted Yu commented on HBASE-4240: --- I am not sure how useful it is for testRandomizer() to test randomize() which resides only in the same test. I was saying that randomize() and testRandomizer() should both be removed. Allow Loadbalancer to be pluggable. --- Key: HBASE-4240 URL: https://issues.apache.org/jira/browse/HBASE-4240 Project: HBase Issue Type: New Feature Components: master Affects Versions: 0.94.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.94.0 Attachments: HBASE-4240-0.patch, HBASE-4240-1.patch, HBASE-4240-2.patch Everyone seems to want something different from a load balancer. People want low latency, simplicity, and total control. It seems like at some point the load balancer can't be all things to all people. Something akin to what hadoop JT's pluggable scheduler seems like it will enable all solutions without making the code much more complex. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4071) Data GC: Remove all versions TTL EXCEPT the last written version
[ https://issues.apache.org/jira/browse/HBASE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089266#comment-13089266 ] jirapos...@reviews.apache.org commented on HBASE-4071: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1582/ --- (Updated 2011-08-23 04:53:14.501780) Review request for hbase, Todd Lipcon, Michael Stack, and Ian Varley. Changes --- More line length and comment fixes. Summary --- Allow enforcing a minimum number of versions when TTL is enable for a store. The GC logic for both versions and TTL is unified inside the ColumnTrackers. This addresses bug HBASE-4071. https://issues.apache.org/jira/browse/HBASE-4071 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/ruby/hbase/admin.rb 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java 1160440 Diff: https://reviews.apache.org/r/1582/diff Testing --- Ran all tests. I get error (not failures) from two: TestDistributedLogSplitting and TestHTablePool. Both fail with or without my changes. New tests: TestMinVersions. Thanks, Lars Data GC: Remove all versions TTL EXCEPT the last written version -- Key: HBASE-4071 URL: https://issues.apache.org/jira/browse/HBASE-4071 Project: HBase Issue Type: New Feature Reporter: stack Attachments: MinVersions.diff We were chatting today about our backup cluster. What we want is to be able to restore the dataset from any point of time but only within a limited timeframe -- say one week. Thereafter, if the versions are older than one week, rather than as we do with TTL where we let go of all versions older than TTL, instead, let go of all versions EXCEPT the last one written. So, its like versions==1 when TTL one week. We want to allow that if an error is caught within a week of its happening -- user mistakenly removes a critical table -- then we'll be able to restore up the the moment just before catastrophe hit otherwise, we keep one version only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4071) Data GC: Remove all versions TTL EXCEPT the last written version
[ https://issues.apache.org/jira/browse/HBASE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089268#comment-13089268 ] jirapos...@reviews.apache.org commented on HBASE-4071: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1582/ --- (Updated 2011-08-23 04:55:59.908653) Review request for hbase, Todd Lipcon, Michael Stack, and Jonathan Gray. Summary --- Allow enforcing a minimum number of versions when TTL is enable for a store. The GC logic for both versions and TTL is unified inside the ColumnTrackers. This addresses bug HBASE-4071. https://issues.apache.org/jira/browse/HBASE-4071 Diffs - http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/ruby/hbase/admin.rb 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java 1160440 Diff: https://reviews.apache.org/r/1582/diff Testing --- Ran all tests. I get error (not failures) from two: TestDistributedLogSplitting and TestHTablePool. Both fail with or without my changes. New tests: TestMinVersions. Thanks, Lars Data GC: Remove all versions TTL EXCEPT the last written version -- Key: HBASE-4071 URL: https://issues.apache.org/jira/browse/HBASE-4071 Project: HBase Issue Type: New Feature Reporter: stack Attachments: MinVersions.diff We were chatting today about our backup cluster. What we want is to be able to restore the dataset from any point of time but only within a limited timeframe -- say one week. Thereafter, if the versions are older than one week, rather than as we do with TTL where we let go of all versions older than TTL, instead, let go of all versions EXCEPT the last one written. So, its like versions==1 when TTL one week. We want to allow that if an error is caught within a week of its happening -- user mistakenly removes a critical table -- then we'll be able to restore up the the moment just before catastrophe hit otherwise, we keep one version only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4242) Add documentation for HBASE-4071
Add documentation for HBASE-4071 Key: HBASE-4242 URL: https://issues.apache.org/jira/browse/HBASE-4242 Project: HBase Issue Type: Sub-task Components: regionserver Affects Versions: 0.92.0 Reporter: Lars Hofhansl Priority: Minor Add documentation for min versions to the HBase book. I'll add patch soon. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4240) Allow Loadbalancer to be pluggable.
[ https://issues.apache.org/jira/browse/HBASE-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-4240: - Attachment: HBASE-4240-3.patch yeah no I idea what I thought that test method was testing. Allow Loadbalancer to be pluggable. --- Key: HBASE-4240 URL: https://issues.apache.org/jira/browse/HBASE-4240 Project: HBase Issue Type: New Feature Components: master Affects Versions: 0.94.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.94.0 Attachments: HBASE-4240-0.patch, HBASE-4240-1.patch, HBASE-4240-2.patch, HBASE-4240-3.patch Everyone seems to want something different from a load balancer. People want low latency, simplicity, and total control. It seems like at some point the load balancer can't be all things to all people. Something akin to what hadoop JT's pluggable scheduler seems like it will enable all solutions without making the code much more complex. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4071) Data GC: Remove all versions TTL EXCEPT the last written version
[ https://issues.apache.org/jira/browse/HBASE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089273#comment-13089273 ] jirapos...@reviews.apache.org commented on HBASE-4071: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1582/#review1596 --- http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java https://reviews.apache.org/r/1582/#comment3603 Javadoc is also needed for this new test. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java https://reviews.apache.org/r/1582/#comment3604 Did you mean 'getClosestRowBefore() can get' - Ted On 2011-08-23 04:55:59, Lars Hofhansl wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1582/ bq. --- bq. bq. (Updated 2011-08-23 04:55:59) bq. bq. bq. Review request for hbase, Todd Lipcon, Michael Stack, and Jonathan Gray. bq. bq. bq. Summary bq. --- bq. bq. Allow enforcing a minimum number of versions when TTL is enable for a store. bq. The GC logic for both versions and TTL is unified inside the ColumnTrackers. bq. bq. bq. This addresses bug HBASE-4071. bq. https://issues.apache.org/jira/browse/HBASE-4071 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1160440 bq.http://svn.apache.org/repos/asf/hbase/trunk/src/main/ruby/hbase/admin.rb 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java 1160440 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java 1160440 bq. bq. Diff: https://reviews.apache.org/r/1582/diff bq. bq. bq. Testing bq. --- bq. bq. Ran all tests. I get error (not failures) from two: TestDistributedLogSplitting and TestHTablePool. Both fail with or without my changes. bq. New tests: TestMinVersions. bq. bq. bq. Thanks, bq. bq. Lars bq. bq. Data GC: Remove all versions TTL EXCEPT the last written version -- Key: HBASE-4071 URL: https://issues.apache.org/jira/browse/HBASE-4071 Project: HBase Issue Type: New Feature Reporter: stack Attachments: MinVersions.diff We were chatting today about our backup cluster. What we want is to be able to restore the dataset from any point of time but only within a limited timeframe -- say one week. Thereafter, if the versions are older than one week, rather than as we do with TTL where we let go of all versions older than TTL, instead, let go of all versions EXCEPT the last one written. So, its like versions==1 when TTL one week. We want to allow that if an error is caught within a week of its happening -- user mistakenly removes a critical table -- then we'll be able to restore up the the moment just before catastrophe hit otherwise, we keep one version only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4071) Data GC: Remove all versions TTL EXCEPT the last written version
[ https://issues.apache.org/jira/browse/HBASE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089276#comment-13089276 ] jirapos...@reviews.apache.org commented on HBASE-4071: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1582/ --- (Updated 2011-08-23 05:13:38.495607) Review request for hbase, Todd Lipcon, Michael Stack, and Jonathan Gray. Changes --- Added Javadoc to TestMinVersions. Summary --- Allow enforcing a minimum number of versions when TTL is enable for a store. The GC logic for both versions and TTL is unified inside the ColumnTrackers. This addresses bug HBASE-4071. https://issues.apache.org/jira/browse/HBASE-4071 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/main/ruby/hbase/admin.rb 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java 1160440 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMinVersions.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java 1160440 Diff: https://reviews.apache.org/r/1582/diff Testing --- Ran all tests. I get error (not failures) from two: TestDistributedLogSplitting and TestHTablePool. Both fail with or without my changes. New tests: TestMinVersions. Thanks, Lars Data GC: Remove all versions TTL EXCEPT the last written version -- Key: HBASE-4071 URL: https://issues.apache.org/jira/browse/HBASE-4071 Project: HBase Issue Type: New Feature Reporter: stack Attachments: MinVersions.diff We were chatting today about our backup cluster. What we want is to be able to restore the dataset from any point of time but only within a limited timeframe -- say one week. Thereafter, if the versions are older than one week, rather than as we do with TTL where we let go of all versions older than TTL, instead, let go of all versions EXCEPT the last one written. So, its like versions==1 when TTL one week. We want to allow that if an error is caught within a week of its happening -- user mistakenly removes a critical table -- then we'll be able to restore up the the moment just before catastrophe hit otherwise, we keep one version only. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira