[jira] [Updated] (HBASE-5225) Backport HBASE-3845 -data loss because lastSeqWritten can miss memstore edits
[ https://issues.apache.org/jira/browse/HBASE-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5225: -- Affects Version/s: 0.90.4 Fix Version/s: 0.90.6 Backport HBASE-3845 -data loss because lastSeqWritten can miss memstore edits - Key: HBASE-5225 URL: https://issues.apache.org/jira/browse/HBASE-5225 Project: HBase Issue Type: Bug Affects Versions: 0.90.4 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.90.6 Critical defect. not merged to 0.90. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5225) Backport HBASE-3845 -data loss because lastSeqWritten can miss memstore edits
Backport HBASE-3845 -data loss because lastSeqWritten can miss memstore edits - Key: HBASE-5225 URL: https://issues.apache.org/jira/browse/HBASE-5225 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Critical defect. not merged to 0.90. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5176) AssignmentManager#getRegion: logging nit adds a redundant '+'
[ https://issues.apache.org/jira/browse/HBASE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188352#comment-13188352 ] Hudson commented on HBASE-5176: --- Integrated in HBase-0.92-security #83 (See [https://builds.apache.org/job/HBase-0.92-security/83/]) HBASE-5176 AssignmentManager#getRegion: logging nit adds a redundant '+' (Karthik K) tedyu : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java AssignmentManager#getRegion: logging nit adds a redundant '+' Key: HBASE-5176 URL: https://issues.apache.org/jira/browse/HBASE-5176 Project: HBase Issue Type: Bug Environment: hadoop 1.0.0 , zk 3.4.2 , hbase 0.92.0 rc3 Reporter: Karthik K Assignee: Karthik K Priority: Minor Fix For: 0.94.0, 0.92.1 Attachments: HBASE-5176.patch From the logs of HMaster: 2012-01-10 17:28:24,370 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for -ROOT-,,0.70236052 destination server is + localhost,60020,1326242475275 Was the '+' intended to be there , as part of some token for log verification or just being redundant , w.r.t the following string append ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5153) Add retry logic in HConnectionImplementation#resetZooKeeperTrackers
[ https://issues.apache.org/jira/browse/HBASE-5153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188350#comment-13188350 ] Hudson commented on HBASE-5153: --- Integrated in HBase-0.92-security #83 (See [https://builds.apache.org/job/HBase-0.92-security/83/]) HBASE-5153 revert due to failed Jenkins builds tedyu : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnection.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java Add retry logic in HConnectionImplementation#resetZooKeeperTrackers --- Key: HBASE-5153 URL: https://issues.apache.org/jira/browse/HBASE-5153 Project: HBase Issue Type: Bug Components: client Affects Versions: 0.90.4 Reporter: Jieshan Bean Assignee: Jieshan Bean Fix For: 0.94.0, 0.92.1, 0.90.6 Attachments: 5153-92.txt, 5153-trunk.txt, 5153-trunk.txt, HBASE-5153-V2.patch, HBASE-5153-V3.patch, HBASE-5153-V4-90.patch, HBASE-5153-V5-90.patch, HBASE-5153-V6-90-minorchange.patch, HBASE-5153-V6-90.txt, HBASE-5153-trunk-v2.patch, HBASE-5153-trunk.patch, HBASE-5153.patch, TestResults-hbase5153.out HBASE-4893 is related to this issue. In that issue, we know, if multi-threads share a same connection, once this connection got abort in one thread, the other threads will got a HConnectionManager$HConnectionImplementation@18fb1f7 closed exception. It solve the problem of stale connection can't removed. But the orignal HTable instance cann't be continue to use. The connection in HTable should be recreated. Actually, there's two aproach to solve this: 1. In user code, once catch an IOE, close connection and re-create HTable instance. We can use this as a workaround. 2. In HBase Client side, catch this exception, and re-create connection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5223) TestMetaReaderEditor is missing call to CatalogTracker.stop()
[ https://issues.apache.org/jira/browse/HBASE-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188351#comment-13188351 ] Hudson commented on HBASE-5223: --- Integrated in HBase-0.92-security #83 (See [https://builds.apache.org/job/HBase-0.92-security/83/]) HBASE-5223 TestMetaReaderEditor is missing call to CatalogTracker.stop() HBASE-5223 TestMetaReaderEditor is missing call to CatalogTracker.stop() tedyu : Files : * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java tedyu : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java TestMetaReaderEditor is missing call to CatalogTracker.stop() - Key: HBASE-5223 URL: https://issues.apache.org/jira/browse/HBASE-5223 Project: HBase Issue Type: Test Reporter: Zhihong Yu Fix For: 0.94.0, 0.92.1 Attachments: 5223.txt I noticed that TestMetaReaderEditor hung on 0.92 Jenkins builds - see https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/249/console It turns out that CatalogTracker.stop() is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5216) [book] book.xml - added more in Arch when to use hbase section
[ https://issues.apache.org/jira/browse/HBASE-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188382#comment-13188382 ] Hudson commented on HBASE-5216: --- Integrated in HBase-TRUNK-security #81 (See [https://builds.apache.org/job/HBase-TRUNK-security/81/]) hbase-5216 book.xml - added detail on Arch when to use hbase section [book] book.xml - added more in Arch when to use hbase section Key: HBASE-5216 URL: https://issues.apache.org/jira/browse/HBASE-5216 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: book_hbase_5216.xml.patch book.xml - Arch chapter, added an additional paragraph in the when to use hbase section. This is based on a lengthy dist-list conversation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5203) Group atomic put/delete operation into a single WALEdit to handle region server failures.
[ https://issues.apache.org/jira/browse/HBASE-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188383#comment-13188383 ] Hudson commented on HBASE-5203: --- Integrated in HBase-TRUNK-security #81 (See [https://builds.apache.org/job/HBase-TRUNK-security/81/]) HBASE-5203 Group atomic put/delete operation into a single WALEdit to handle region server failures. (Lars H) larsh : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Delete.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java Group atomic put/delete operation into a single WALEdit to handle region server failures. - Key: HBASE-5203 URL: https://issues.apache.org/jira/browse/HBASE-5203 Project: HBase Issue Type: Sub-task Components: client, coprocessors, regionserver Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 5203-v3.txt, 5203.txt HBASE-3584 does not not provide fully atomic operation in case of region server failures (see explanation there). What should happen is that either (1) all edits are applied via a single WALEdit, or (2) the WALEdits are applied in async mode and then sync'ed together. For #1 it is not clear whether it is advisable to manage multiple *different* operations (Put/Delete) via a single WAL edit. A quick check reveals that WAL replay on region startup would work, but that replication would need to be adapted. The refactoring needed would be non-trivial. #2 Might actually not work, as another operation could request sync'ing a later edit and hence flush these entries out as well. Addendum: The attached patch implements #1 and fixes replication to be able to deal with different operations being grouped in one WALEdit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5201) Utilize TThreadedSelectorServer and remove redundant code in ThriftServer and HRegionThriftServer
[ https://issues.apache.org/jira/browse/HBASE-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188385#comment-13188385 ] Hudson commented on HBASE-5201: --- Integrated in HBase-TRUNK-security #81 (See [https://builds.apache.org/job/HBase-TRUNK-security/81/]) HBASE-5201 Add new files HBASE-5201 Utilize TThreadedSelectorServer and remove redundant code in ThriftServer and HRegionThriftServer (Scott Chen) tedyu : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/HThreadedSelectorServerArgs.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java tedyu : Files : * /hbase/trunk/pom.xml * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java Utilize TThreadedSelectorServer and remove redundant code in ThriftServer and HRegionThriftServer - Key: HBASE-5201 URL: https://issues.apache.org/jira/browse/HBASE-5201 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.94.0 Attachments: HBASE-5201-v2.txt, HBASE-5201-v3.txt, HBASE-5201-v4.txt, HBASE-5201.txt TThreadedSelectorServer is good for RPC-heavy situation because IO are not limited to one CPU. See https://issues.apache.org/jira/browse/Thrift-1167 I am porting the related classes form thrift trunk (it is not there in thrift-0.7.0). There are lots of repeat codes in ThriftServer and HRegionThriftServer. These codes are now moved to a Runnable called ThriftServerRunner. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5153) Add retry logic in HConnectionImplementation#resetZooKeeperTrackers
[ https://issues.apache.org/jira/browse/HBASE-5153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188384#comment-13188384 ] Hudson commented on HBASE-5153: --- Integrated in HBase-TRUNK-security #81 (See [https://builds.apache.org/job/HBase-TRUNK-security/81/]) HBASE-5153 revert due to failed Jenkins builds tedyu : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ClosedConnectionException.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnection.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/MasterSchemaChangeTracker.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/SchemaChangeTracker.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithAbort.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterCoprocessorExceptionWithRemove.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java Add retry logic in HConnectionImplementation#resetZooKeeperTrackers --- Key: HBASE-5153 URL: https://issues.apache.org/jira/browse/HBASE-5153 Project: HBase Issue Type: Bug Components: client Affects Versions: 0.90.4 Reporter: Jieshan Bean Assignee: Jieshan Bean Fix For: 0.94.0, 0.92.1, 0.90.6 Attachments: 5153-92.txt, 5153-trunk.txt, 5153-trunk.txt, HBASE-5153-V2.patch, HBASE-5153-V3.patch, HBASE-5153-V4-90.patch, HBASE-5153-V5-90.patch, HBASE-5153-V6-90-minorchange.patch, HBASE-5153-V6-90.txt, HBASE-5153-trunk-v2.patch, HBASE-5153-trunk.patch, HBASE-5153.patch, TestResults-hbase5153.out HBASE-4893 is related to this issue. In that issue, we know, if multi-threads share a same connection, once this connection got abort in one thread, the other threads will got a HConnectionManager$HConnectionImplementation@18fb1f7 closed exception. It solve the problem of stale connection can't removed. But the orignal HTable instance cann't be continue to use. The connection in HTable should be recreated. Actually, there's two aproach to solve this: 1. In user code, once catch an IOE, close connection and re-create HTable instance. We can use this as a workaround. 2. In HBase Client side, catch this exception, and re-create connection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1621) merge tool should work on online cluster, but disabled table
[ https://issues.apache.org/jira/browse/HBASE-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188386#comment-13188386 ] Hudson commented on HBASE-1621: --- Integrated in HBase-TRUNK-security #81 (See [https://builds.apache.org/job/HBase-TRUNK-security/81/]) [book] book.xml, ops_mgt.xml additional clarification that compaction does not do region merging also, added link to script in HBASE-1621 in Ops Mgt chapter. dmeil : Files : * /hbase/trunk/src/docbkx/book.xml * /hbase/trunk/src/docbkx/ops_mgt.xml merge tool should work on online cluster, but disabled table Key: HBASE-1621 URL: https://issues.apache.org/jira/browse/HBASE-1621 Project: HBase Issue Type: Bug Reporter: ryan rawson Assignee: stack Fix For: 0.94.0 Attachments: 1621-trunk.txt, HBASE-1621-v2.patch, HBASE-1621.patch, hbase-onlinemerge.patch, online_merge.rb taking down the entire cluster to merge 2 regions is a pain, i dont see why the table or regions specifically couldnt be taken offline, then merged then brought back up. this might need a new API to the regionservers so they can take direction from not just the master. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5220) [book] troubleshooting.xml - adding information about 'zkcli' to Troubleshooting chapter
[ https://issues.apache.org/jira/browse/HBASE-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188388#comment-13188388 ] Hudson commented on HBASE-5220: --- Integrated in HBase-TRUNK-security #81 (See [https://builds.apache.org/job/HBase-TRUNK-security/81/]) hbase-5220 troubleshooting.xml - zkcli info [book] troubleshooting.xml - adding information about 'zkcli' to Troubleshooting chapter Key: HBASE-5220 URL: https://issues.apache.org/jira/browse/HBASE-5220 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: troubleshooting_hbase_5220.xml.patch troubleshooting.xml * added entry in 'builtin' tools section on zkcli * added link in trouble.zookeeper.general to the above entry This utility came up on the dist-list recently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5176) AssignmentManager#getRegion: logging nit adds a redundant '+'
[ https://issues.apache.org/jira/browse/HBASE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188389#comment-13188389 ] Hudson commented on HBASE-5176: --- Integrated in HBase-TRUNK-security #81 (See [https://builds.apache.org/job/HBase-TRUNK-security/81/]) HBASE-5176 AssignmentManager#getRegion: logging nit adds a redundant '+' (Karthik K) tedyu : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java AssignmentManager#getRegion: logging nit adds a redundant '+' Key: HBASE-5176 URL: https://issues.apache.org/jira/browse/HBASE-5176 Project: HBase Issue Type: Bug Environment: hadoop 1.0.0 , zk 3.4.2 , hbase 0.92.0 rc3 Reporter: Karthik K Assignee: Karthik K Priority: Minor Fix For: 0.94.0, 0.92.1 Attachments: HBASE-5176.patch From the logs of HMaster: 2012-01-10 17:28:24,370 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for -ROOT-,,0.70236052 destination server is + localhost,60020,1326242475275 Was the '+' intended to be there , as part of some token for log verification or just being redundant , w.r.t the following string append ? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5218) [book] book.xml - added Arch/Hfile, added link to HFile v2 info in appendix
[ https://issues.apache.org/jira/browse/HBASE-5218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188387#comment-13188387 ] Hudson commented on HBASE-5218: --- Integrated in HBase-TRUNK-security #81 (See [https://builds.apache.org/job/HBase-TRUNK-security/81/]) hbase-5218 book.xml - Arch/HFile, added link to HFile v2 info in appendix from this section. [book] book.xml - added Arch/Hfile, added link to HFile v2 info in appendix --- Key: HBASE-5218 URL: https://issues.apache.org/jira/browse/HBASE-5218 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Trivial Attachments: book_hbase_5218.xml.patch Stack asked me to do this in December: added link in Arch/HFile to the HFile v2 information in the appendix. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5208) Allow setting Scan start/stop row individually in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188406#comment-13188406 ] Nicholas Telford commented on HBASE-5208: - Regarding test length, without my additions (i.e. clean trunk) the tests take a very long time. Most of the time is spent doing the original tests as they spin up 11 MapReduce jobs. I imagine running the tests in parallel might improve things, but I haven't tested that. My additional test adds another MapReduce job, so it will increase the test length, but not substantially. Allow setting Scan start/stop row individually in TableInputFormat -- Key: HBASE-5208 URL: https://issues.apache.org/jira/browse/HBASE-5208 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Nicholas Telford Priority: Minor Attachments: HBASE-5208-001.txt, HBASE-5208-002.txt, HBASE-5208-003.txt, HBASE-5208-004.txt Currently, TableInputFormat initializes a serialized Scan from hbase.mapreduce.scan. Alternatively, it will instantiate a new Scan using properties defined in hbase.mapreduce.scan.*. However, of these properties the start row and stop row (arguably the most pertinent) are missing. TableInputFormat should permit the specification of a start/stop row as with the other fields using a new pair of properties: hbase.mapreduce.scan.row.start and hbase.mapreduce.scan.row.end The primary use-case for this is to permit Oozie and other job management tools that can't call TableMapReduceUtil.initTableMapperJob() to operate on a contiguous subset of rows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
[
https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188413#comment-13188413
]
Zhihong Yu commented on HBASE-5179:
---
{code}
+ * Dead servers under processing by the ServerShutdownHander. Map of
+ * ServerType to set of being processed dead servers
+ */
+ private final MapServerType, SetServerName deadServersUnderProcessing =
new HashMapServerType, SetServerName();
{code}
The second line of javadoc should read: ' to set of dead servers being
processed'
Please wrap long line.
I think we should pre-create HashSet's for the three ServerType's in DeadServer
ctor.
This way we can omit runtime checks such as:
{code}
+if (deadNormalServersBeingProcessed == null) {
+ deadNormalServersBeingProcessed = new HashSetServerName();
+ deadServersUnderProcessing.put(ServerType.NORMAL,
+ deadNormalServersBeingProcessed);
+}
{code}
{code}
+ * @param assuredRootServer true if it's dead server carrying root certainly.
+ * @param assuredMetaServer true if it's dead server carrying meta certainly.
{code}
How about naming the parameters definitiveRootServer and definitiveMetaServer ?
(Change certainly to definitely).
We should use low value for hbase.master.wait.on.regionservers.timeout so
that the new test case doesn't take too long.
Concurrent processing of processFaileOver and ServerShutdownHandler may cause
region to be assigned before log splitting is completed, causing data loss
Key: HBASE-5179
URL: https://issues.apache.org/jira/browse/HBASE-5179
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.2
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.92.0, 0.94.0, 0.90.6
Attachments: 5179-90.txt, 5179-90v2.patch, 5179-90v3.patch,
5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch,
5179-90v8.patch, 5179-90v9.patch, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt,
hbase-5179.patch, hbase-5179v10.patch, hbase-5179v5.patch,
hbase-5179v6.patch, hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch
If master's processing its failover and ServerShutdownHandler's processing
happen concurrently, it may appear following case.
1.master completed splitLogAfterStartup()
2.RegionserverA restarts, and ServerShutdownHandler is processing.
3.master starts to rebuildUserRegions, and RegionserverA is considered as
dead server.
4.master starts to assign regions of RegionserverA because it is a dead
server by step3.
However, when doing step4(assigning region), ServerShutdownHandler may be
doing split log, Therefore, it may cause data loss.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5201) Utilize TThreadedSelectorServer and remove redundant code in ThriftServer and HRegionThriftServer
[ https://issues.apache.org/jira/browse/HBASE-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated HBASE-5201: -- Release Note: New config parameters: hbase.thrift.selector.threads Number of selector threads for reading and writing socket hbase.thrift.worker.threads Number of threads for processing the thrift calls hbase.thrift.stop.timeout.seconds Time to wait for server to stop gracefully hbase.thrift.accept.queue.size.per.selector Maximum number of accepted elements per selector hbase.thrift.accept.policy The strategy for handling new accepted connections was: New config parameters: hbase.thrift.selector.threads Number of selector threads for reading and writing socket hbase.thrift.worker.threads Number of threads for processing the thrift calls hbase.thrift.stop.timeout.seconds Time to wait for server to stop gracefully hbase.thrift.accept.queue.size.per.selector Maximum number of accepted elements per selector Utilize TThreadedSelectorServer and remove redundant code in ThriftServer and HRegionThriftServer - Key: HBASE-5201 URL: https://issues.apache.org/jira/browse/HBASE-5201 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.94.0 Attachments: HBASE-5201-v2.txt, HBASE-5201-v3.txt, HBASE-5201-v4.txt, HBASE-5201.txt TThreadedSelectorServer is good for RPC-heavy situation because IO are not limited to one CPU. See https://issues.apache.org/jira/browse/Thrift-1167 I am porting the related classes form thrift trunk (it is not there in thrift-0.7.0). There are lots of repeat codes in ThriftServer and HRegionThriftServer. These codes are now moved to a Runnable called ThriftServerRunner. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5201) Utilize TThreadedSelectorServer and remove redundant code in ThriftServer and HRegionThriftServer
[ https://issues.apache.org/jira/browse/HBASE-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188445#comment-13188445 ] Scott Chen commented on HBASE-5201: --- Added one more config to the release note. Utilize TThreadedSelectorServer and remove redundant code in ThriftServer and HRegionThriftServer - Key: HBASE-5201 URL: https://issues.apache.org/jira/browse/HBASE-5201 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Scott Chen Assignee: Scott Chen Fix For: 0.94.0 Attachments: HBASE-5201-v2.txt, HBASE-5201-v3.txt, HBASE-5201-v4.txt, HBASE-5201.txt TThreadedSelectorServer is good for RPC-heavy situation because IO are not limited to one CPU. See https://issues.apache.org/jira/browse/Thrift-1167 I am porting the related classes form thrift trunk (it is not there in thrift-0.7.0). There are lots of repeat codes in ThriftServer and HRegionThriftServer. These codes are now moved to a Runnable called ThriftServerRunner. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5226) [book] troubleshooting.xml - add client slowdown when calling admin APIs issue to Client section
[book] troubleshooting.xml - add client slowdown when calling admin APIs issue to Client section Key: HBASE-5226 URL: https://issues.apache.org/jira/browse/HBASE-5226 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor troubleshooting.xml - adding client slowdown when calling admin APIs issue (HBASE-5073) to the Client section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5226) [book] troubleshooting.xml - add client slowdown when calling admin APIs issue to Client section
[ https://issues.apache.org/jira/browse/HBASE-5226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-5226: - Status: Patch Available (was: Open) [book] troubleshooting.xml - add client slowdown when calling admin APIs issue to Client section Key: HBASE-5226 URL: https://issues.apache.org/jira/browse/HBASE-5226 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: troubleshooting_hbase_5226.xml.patch troubleshooting.xml - adding client slowdown when calling admin APIs issue (HBASE-5073) to the Client section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5226) [book] troubleshooting.xml - add client slowdown when calling admin APIs issue to Client section
[ https://issues.apache.org/jira/browse/HBASE-5226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-5226: - Attachment: troubleshooting_hbase_5226.xml.patch [book] troubleshooting.xml - add client slowdown when calling admin APIs issue to Client section Key: HBASE-5226 URL: https://issues.apache.org/jira/browse/HBASE-5226 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: troubleshooting_hbase_5226.xml.patch troubleshooting.xml - adding client slowdown when calling admin APIs issue (HBASE-5073) to the Client section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5226) [book] troubleshooting.xml - add client slowdown when calling admin APIs issue to Client section
[ https://issues.apache.org/jira/browse/HBASE-5226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-5226: - Resolution: Fixed Status: Resolved (was: Patch Available) [book] troubleshooting.xml - add client slowdown when calling admin APIs issue to Client section Key: HBASE-5226 URL: https://issues.apache.org/jira/browse/HBASE-5226 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: troubleshooting_hbase_5226.xml.patch troubleshooting.xml - adding client slowdown when calling admin APIs issue (HBASE-5073) to the Client section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4805) Allow better control of resource consumption in HTable
[ https://issues.apache.org/jira/browse/HBASE-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188541#comment-13188541 ] Krystian Nowak commented on HBASE-4805: --- Do you plan to backport it to 0.90.x branch as well or I should rather wait for 0.92.x+? Cheers, Krystian Allow better control of resource consumption in HTable -- Key: HBASE-4805 URL: https://issues.apache.org/jira/browse/HBASE-4805 Project: HBase Issue Type: Bug Components: client Affects Versions: 0.92.0, 0.94.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.92.0, 0.94.0 Attachments: 4805-v2.txt, 4805-v3.txt, 4805-v4.txt, 4805-v5.txt, 4805.txt From some internal discussions at Salesforce we concluded that we need better control over the resources (mostly threads) consumed by HTable when used in a AppServer with many client threads. Since HTable is not thread safe, the only options are cache them (in a custom thread local or using HTablePool) or to create them on-demand. I propose a simple change: Add a new constructor to HTable that takes an optional ExecutorService and HConnection instance. That would make HTable a pretty lightweight object and we would manage the ES and HC separately. I'll upload a patch a soon to get some feedback. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5227) [book] added warning about using RowLocks
[book] added warning about using RowLocks - Key: HBASE-5227 URL: https://issues.apache.org/jira/browse/HBASE-5227 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor book.xml * Arch/Client - added entry discouraging users from RowLock, with link to outstanding Jira on removing the feature entirely. troubleshooting.xml * RS Runtime section. Under existing RS Hanging entry added a paragraph pointing to the newly creatd rowlock entry in the Arch/Client section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5227) [book] added warning about using RowLocks
[ https://issues.apache.org/jira/browse/HBASE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-5227: - Resolution: Fixed Status: Resolved (was: Patch Available) [book] added warning about using RowLocks - Key: HBASE-5227 URL: https://issues.apache.org/jira/browse/HBASE-5227 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_hbase_5227.patch book.xml * Arch/Client - added entry discouraging users from RowLock, with link to outstanding Jira on removing the feature entirely. troubleshooting.xml * RS Runtime section. Under existing RS Hanging entry added a paragraph pointing to the newly creatd rowlock entry in the Arch/Client section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5227) [book] added warning about using RowLocks
[ https://issues.apache.org/jira/browse/HBASE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-5227: - Attachment: docbkx_hbase_5227.patch [book] added warning about using RowLocks - Key: HBASE-5227 URL: https://issues.apache.org/jira/browse/HBASE-5227 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_hbase_5227.patch book.xml * Arch/Client - added entry discouraging users from RowLock, with link to outstanding Jira on removing the feature entirely. troubleshooting.xml * RS Runtime section. Under existing RS Hanging entry added a paragraph pointing to the newly creatd rowlock entry in the Arch/Client section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5227) [book] added warning about using RowLocks
[ https://issues.apache.org/jira/browse/HBASE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-5227: - Status: Patch Available (was: Open) [book] added warning about using RowLocks - Key: HBASE-5227 URL: https://issues.apache.org/jira/browse/HBASE-5227 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: docbkx_hbase_5227.patch book.xml * Arch/Client - added entry discouraging users from RowLock, with link to outstanding Jira on removing the feature entirely. troubleshooting.xml * RS Runtime section. Under existing RS Hanging entry added a paragraph pointing to the newly creatd rowlock entry in the Arch/Client section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
[ https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5179: -- Attachment: 5179-v11.txt Patch v11 simplifies the logic in DeadServer.java Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss Key: HBASE-5179 URL: https://issues.apache.org/jira/browse/HBASE-5179 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.6 Attachments: 5179-90.txt, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch If master's processing its failover and ServerShutdownHandler's processing happen concurrently, it may appear following case. 1.master completed splitLogAfterStartup() 2.RegionserverA restarts, and ServerShutdownHandler is processing. 3.master starts to rebuildUserRegions, and RegionserverA is considered as dead server. 4.master starts to assign regions of RegionserverA because it is a dead server by step3. However, when doing step4(assigning region), ServerShutdownHandler may be doing split log, Therefore, it may cause data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3373) Allow regions to be load-balanced by table
[ https://issues.apache.org/jira/browse/HBASE-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-3373: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Allow regions to be load-balanced by table -- Key: HBASE-3373 URL: https://issues.apache.org/jira/browse/HBASE-3373 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.20.6 Reporter: Ted Yu Fix For: 0.94.0 Attachments: 3373.txt, HbaseBalancerTest2.java From our experience, cluster can be well balanced and yet, one table's regions may be badly concentrated on few region servers. For example, one table has 839 regions (380 regions at time of table creation) out of which 202 are on one server. It would be desirable for load balancer to distribute regions for specified tables evenly across the cluster. Each of such tables has number of regions many times the cluster size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4542) add filter info to slow query logging
[
https://issues.apache.org/jira/browse/HBASE-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HBASE-4542:
---
Attachment: D1263.2.patch
zhiqiu requested code review of [jira] [HBase-4542] [89-fb] Add filter info to
slow query logging.
Reviewers: Kannan, madhuvaidya, mbautin, JIRA
Slow opertaion log does not provide enough information when a filter is
present. The followings are done to add the filter info:
1) Added toString() method for filters inheriting FilterBase, this
affect 22 filters and their subclasses. The info added includes the
filter's name and its members. For example, for TimestampsFilter, we'll
output its class name as well as the defined timestamps.
2) Added a field 'filter' in Get::toMap() and
Scan::toMap() to enable the logging of filter info.
TEST PLAN
1. Run and passed unit-tests to make sure it does not break things
2. Run kannan's script to trigger the slow operation logging, checked
for each filter to make sure the filter info was logged. To be more
detailed, the output log are as following (only 'filter' filed is put
here for ease of reading):
* filter:TimestampsFilter (3/3): [2, 3, 5]
* filter:TimestampsFilter (5/6): [2, 3, 5, 7, 11]
* filter:ColumnPrefixFilter col2
* filter:ColumnRangeFilter [col2a, col2b]
* filter:ColumnCountGetFilter 8
* filter:ColumnPaginationFilter (4, 4)
* filter:InclusiveStopFilter row
* filter:PrefixFilter row
* filter:PageFilter 1
* filter:SkipFilter TimestampsFilter (1/1): [1000]
* filter:WhileMatchFilter TimestampsFilter (3/3): [2, 3, 5]
* filter:KeyOnlyFilter
* filter:FirstKeyOnlyFilter
* filter:MultipleColumnPrefixFilter (3/3): [a, b, c]
* filter:DependentColumnFilter (family, qualifier, true, LESS, value)
* filter:FamilyFilter (LESS, value)
* filter:QualifierFilter (LESS, value)
* filter:RowFilter (LESS, value)
* filter:ValueFilter (LESS, value)
* filter:KeyOnlyFilter
* filter:FirstKeyOnlyFilter
* filter:SingleColumnValueFilter (family, qualifier, EQUAL, value)
* filter:SingleColumnValueExcludeFilter (family, qualifier, EQUAL,
value)
* filter:FilterList AND (2/2): [KeyOnlyFilter, FirstKeyOnlyFilter]
Please check ~zhiqiu/Codes/scripts/testFilter.rb for the testing script.
3. Added unit test cases to TestOperation to verify the filters'
toString() method works well.
Revert Plan:
Tags:
REVISION DETAIL
https://reviews.facebook.net/D1263
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/client/Get.java
src/main/java/org/apache/hadoop/hbase/client/Scan.java
src/main/java/org/apache/hadoop/hbase/filter/ColumnCountGetFilter.java
src/main/java/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.java
src/main/java/org/apache/hadoop/hbase/filter/ColumnPrefixFilter.java
src/main/java/org/apache/hadoop/hbase/filter/ColumnRangeFilter.java
src/main/java/org/apache/hadoop/hbase/filter/CompareFilter.java
src/main/java/org/apache/hadoop/hbase/filter/DependentColumnFilter.java
src/main/java/org/apache/hadoop/hbase/filter/FilterBase.java
src/main/java/org/apache/hadoop/hbase/filter/FilterList.java
src/main/java/org/apache/hadoop/hbase/filter/InclusiveStopFilter.java
src/main/java/org/apache/hadoop/hbase/filter/MultipleColumnPrefixFilter.java
src/main/java/org/apache/hadoop/hbase/filter/PageFilter.java
src/main/java/org/apache/hadoop/hbase/filter/PrefixFilter.java
src/main/java/org/apache/hadoop/hbase/filter/SingleColumnValueFilter.java
src/main/java/org/apache/hadoop/hbase/filter/SkipFilter.java
src/main/java/org/apache/hadoop/hbase/filter/TimestampsFilter.java
src/main/java/org/apache/hadoop/hbase/filter/WhileMatchFilter.java
src/test/java/org/apache/hadoop/hbase/client/TestOperation.java
add filter info to slow query logging
-
Key: HBASE-4542
URL: https://issues.apache.org/jira/browse/HBASE-4542
Project: HBase
Issue Type: Improvement
Reporter: Kannan Muthukkaruppan
Assignee: Madhuwanti Vaidya
Attachments: D1263.2.patch
Slow query log doesn't report filters in effect.
For example:
{code}
(operationTooSlow): \
{processingtimems:3468,client:10.138.43.206:40035,timeRange:
[0,9223372036854775807],\
starttimems:1317772005821,responsesize:42411, \
class:HRegionServer,table:myTable,families:{CF1:ALL]},\
row:6c3b8efa132f0219b7621ed1e5c8c70b,queuetimems:0,\
method:get,totalColumns:1,maxVersions:1,storeLimit:-1}
{code}
the above would suggest that all columns of myTable:CF1 are being requested
for the given row. But in reality there could be filters in effect (such as
ColumnPrefixFilter, ColumnRangeFilter, TimestampsFilter() etc.). We should
enhance the slow query log to capture report this information.
--
This message is automatically
[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
[ https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188599#comment-13188599 ] Lars Hofhansl commented on HBASE-5179: -- v11 lgtm +1 Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss Key: HBASE-5179 URL: https://issues.apache.org/jira/browse/HBASE-5179 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.6 Attachments: 5179-90.txt, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch If master's processing its failover and ServerShutdownHandler's processing happen concurrently, it may appear following case. 1.master completed splitLogAfterStartup() 2.RegionserverA restarts, and ServerShutdownHandler is processing. 3.master starts to rebuildUserRegions, and RegionserverA is considered as dead server. 4.master starts to assign regions of RegionserverA because it is a dead server by step3. However, when doing step4(assigning region), ServerShutdownHandler may be doing split log, Therefore, it may cause data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
[ https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188607#comment-13188607 ] Hadoop QA commented on HBASE-5179: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511011/5179-v11.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -143 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 82 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestImportTsv Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/805//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/805//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/805//console This message is automatically generated. Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss Key: HBASE-5179 URL: https://issues.apache.org/jira/browse/HBASE-5179 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.6 Attachments: 5179-90.txt, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch If master's processing its failover and ServerShutdownHandler's processing happen concurrently, it may appear following case. 1.master completed splitLogAfterStartup() 2.RegionserverA restarts, and ServerShutdownHandler is processing. 3.master starts to rebuildUserRegions, and RegionserverA is considered as dead server. 4.master starts to assign regions of RegionserverA because it is a dead server by step3. However, when doing step4(assigning region), ServerShutdownHandler may be doing split log, Therefore, it may cause data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
[ https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5179: -- Attachment: 5179-v11-92.txt Patch for 0.92 branch Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss Key: HBASE-5179 URL: https://issues.apache.org/jira/browse/HBASE-5179 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.6 Attachments: 5179-90.txt, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch If master's processing its failover and ServerShutdownHandler's processing happen concurrently, it may appear following case. 1.master completed splitLogAfterStartup() 2.RegionserverA restarts, and ServerShutdownHandler is processing. 3.master starts to rebuildUserRegions, and RegionserverA is considered as dead server. 4.master starts to assign regions of RegionserverA because it is a dead server by step3. However, when doing step4(assigning region), ServerShutdownHandler may be doing split log, Therefore, it may cause data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before compaction selection
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5199: --- Attachment: D1311.1.patch Liyin requested code review of [jira][HBASE-5199] Delete out of TTL store files before compaction selection . Reviewers: Kannan, JIRA Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions. In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection. TEST PLAN TestStore REVISION DETAIL https://reviews.facebook.net/D1311 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/regionserver/Store.java src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java Delete out of TTL store files before compaction selection - Key: HBASE-5199 URL: https://issues.apache.org/jira/browse/HBASE-5199 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D1311.1.patch Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions. In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files before compaction selection
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188633#comment-13188633 ] Phabricator commented on HBASE-5199: Kannan has added reviewers to the revision [jira][HBASE-5199] Delete out of TTL store files before compaction selection . Added Reviewers: khemani, aaiyer, Karthik adding a few more reviewers in case someone else can get to this earlier. REVISION DETAIL https://reviews.facebook.net/D1311 Delete out of TTL store files before compaction selection - Key: HBASE-5199 URL: https://issues.apache.org/jira/browse/HBASE-5199 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D1311.1.patch Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions. In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5204) Backward compatibility fixes for 0.92
[
https://issues.apache.org/jira/browse/HBASE-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188632#comment-13188632
]
Benoit Sigoure commented on HBASE-5204:
---
Sorry I found one more problem. In HBASE-3335 another code was inserted in the
middle of the list for {{BitComparator}}. Any chance that we could move this
one to the end too?
Backward compatibility fixes for 0.92
-
Key: HBASE-5204
URL: https://issues.apache.org/jira/browse/HBASE-5204
Project: HBase
Issue Type: Bug
Components: ipc
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Blocker
Labels: backwards-compatibility
Fix For: 0.92.0
Attachments:
0001-Add-some-backward-compatible-support-for-reading-old.patch,
0002-Make-sure-that-a-connection-always-uses-a-protocol.patch,
0003-Change-the-code-used-when-serializing-HTableDescript.patch, 5204-92.txt,
5204-trunk.txt
Attached are 3 patches that are necessary to allow compatibility between
HBase 0.90.x (and previous releases) and HBase 0.92.0.
First of all, I'm well aware that 0.92.0 RC4 has been thumbed up by a lot of
people and would probably wind up being released as 0.92.0 tomorrow, so I
sincerely apologize for creating this issue so late in the process. I spent
a lot of time trying to work around the quirks of 0.92 but once I realized
that with a few very quasi-trivial changes compatibility would be made
significantly easier, I immediately sent these 3 patches to Stack, who
suggested I create this issue.
The first patch is required as without it clients sending a 0.90-style RPC to
a 0.92-style server causes the server to die uncleanly. It seems that 0.92
ships with {{\-XX:OnOutOfMemoryError=kill \-9 %p}}, and when a 0.92 server
fails to deserialize a 0.90-style RPC, it attempts to allocate a large buffer
because it doesn't read fields of 0.90-style RPCs properly. This allocation
attempt immediately triggers an OOME, which causes the JVM to die abruptly of
a {{SIGKILL}}. So whenever a 0.90.x client attempts to connect to HBase, it
kills whichever RS is hosting the {{\-ROOT-}} region.
The second patch fixes a bug introduced by HBASE-2002, which added support
for letting clients specify what protocol they want to speak. If a client
doesn't properly specify what protocol to use, the connection's {{protocol}}
field will be left {{null}}, which causes any subsequent RPC on that
connection to trigger an NPE in the server, even though the connection was
successfully established from the client's point of view. The fix is to
simply give the connection a default protocol, by assuming the client meant
to speak to a RegionServer.
The third patch fixes an oversight that slipped in HBASE-451, where a change
to {{HbaseObjectWritable}} caused all the codes used to serialize
{{Writables}} to shift by one. This was carefully avoided in other changes
such as HBASE-1502, which cleanly removed entries for {{HMsg}} and
{{HMsg[]}}, so I don't think this breakage in HBASE-451 was intended.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4542) add filter info to slow query logging
[
https://issues.apache.org/jira/browse/HBASE-4542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188634#comment-13188634
]
Phabricator commented on HBASE-4542:
mbautin has accepted the revision [jira] [HBase-4542] [89-fb] Add filter info
to slow query logging.
@zhiqiu: looks good!
REVISION DETAIL
https://reviews.facebook.net/D1263
add filter info to slow query logging
-
Key: HBASE-4542
URL: https://issues.apache.org/jira/browse/HBASE-4542
Project: HBase
Issue Type: Improvement
Reporter: Kannan Muthukkaruppan
Assignee: Madhuwanti Vaidya
Attachments: D1263.2.patch
Slow query log doesn't report filters in effect.
For example:
{code}
(operationTooSlow): \
{processingtimems:3468,client:10.138.43.206:40035,timeRange:
[0,9223372036854775807],\
starttimems:1317772005821,responsesize:42411, \
class:HRegionServer,table:myTable,families:{CF1:ALL]},\
row:6c3b8efa132f0219b7621ed1e5c8c70b,queuetimems:0,\
method:get,totalColumns:1,maxVersions:1,storeLimit:-1}
{code}
the above would suggest that all columns of myTable:CF1 are being requested
for the given row. But in reality there could be filters in effect (such as
ColumnPrefixFilter, ColumnRangeFilter, TimestampsFilter() etc.). We should
enhance the slow query log to capture report this information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2600) Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid
[ https://issues.apache.org/jira/browse/HBASE-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188640#comment-13188640 ] jirapos...@reviews.apache.org commented on HBASE-2600: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3466/ --- (Updated 2012-01-18 19:19:27.118173) Review request for hbase, Michael Stack and Lars Hofhansl. Changes --- Fixed the spacing and got most of stack's comments in. Summary --- This is an idea that Ryan and I have been kicking around on and off for a while now. If regionnames were made of tablename+endrow instead of tablename+startrow, then in the metatables, doing a search for the region that contains the wanted row, we'd just have to open a scanner using passed row and the first row found by the scan would be that of the region we need (If offlined parent, we'd have to scan to the next row). If we redid the meta tables in this format, we'd be using an access that is natural to hbase, a scan as opposed to the perverse, expensive getClosestRowBefore we currently have that has to walk backward in meta finding a containing region. This issue is about changing the way we name regions. If we were using scans, prewarming client cache would be near costless (as opposed to what we'll currently have to do which is first a getClosestRowBefore and then a scan from the closestrowbefore forward). Converting to the new method, we'd have to run a migration on startup changing the content in meta. Up to this, the randomid component of a region name has been the timestamp of region creation. HBASE-2531 32-bit encoding of regionnames waaay too susceptible to hash clashes proposes changing the randomid so that it contains actual name of the directory in the filesystem that hosts the region. If we had this in place, I think it would help with the migration to this new way of doing the meta because as is, the region name in fs is a hash of regionname... changing the format of the regionname would mean we generate a different hash... so we'd need hbase-2531 to be in place before we could do this change. This addresses bug HBASE-2600. https://issues.apache.org/jira/browse/HBASE-2600 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 74cb821 src/main/java/org/apache/hadoop/hbase/HConstants.java 8370ef8 src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java 133759d src/main/java/org/apache/hadoop/hbase/KeyValue.java be7e2d8 src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java e5e60a8 src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 88c381f src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java d857538 src/main/java/org/apache/hadoop/hbase/client/HTable.java 57605e6 src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java 784fdc2 src/main/java/org/apache/hadoop/hbase/client/HTablePool.java a5c198f src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f0c6828 src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java 7a7b896 src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java b1b5a78 src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java c0a4184 src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 0431444 src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java 4307d89 src/main/java/org/apache/hadoop/hbase/regionserver/GetClosestRowBeforeTracker.java 3a26bbb src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c7cc402 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 5cb606f src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 50e7fe0 src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java a3850e5 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 636e533 src/main/java/org/apache/hadoop/hbase/rest/RegionsResource.java bf85bc1 src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java 3919985 src/main/java/org/apache/hadoop/hbase/rest/model/TableRegionModel.java 67e7a04 src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java b6a6349 src/main/java/org/apache/hadoop/hbase/thrift/generated/Hbase.java 9e31c61 src/main/resources/org/apache/hadoop/hbase/thrift/Hbase.thrift 5821d31 src/test/java/org/apache/hadoop/hbase/TestKeyValue.java dc4ee8d src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 95ab8e6 src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java dacb936 src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java 5f97167
[jira] [Updated] (HBASE-2600) Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid
[ https://issues.apache.org/jira/browse/HBASE-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Newman updated HBASE-2600: --- Attachment: 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v9.patch Fixed spacing. Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid Key: HBASE-2600 URL: https://issues.apache.org/jira/browse/HBASE-2600 Project: HBase Issue Type: Bug Reporter: stack Assignee: Alex Newman Attachments: 0001-Changed-regioninfo-format-to-use-endKey-instead-of-s.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v2.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v4.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v6.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v7.2.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8.1, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v9.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen.patch, 2600-trunk-01-17.txt, jenkins.pdf This is an idea that Ryan and I have been kicking around on and off for a while now. If regionnames were made of tablename+endrow instead of tablename+startrow, then in the metatables, doing a search for the region that contains the wanted row, we'd just have to open a scanner using passed row and the first row found by the scan would be that of the region we need (If offlined parent, we'd have to scan to the next row). If we redid the meta tables in this format, we'd be using an access that is natural to hbase, a scan as opposed to the perverse, expensive getClosestRowBefore we currently have that has to walk backward in meta finding a containing region. This issue is about changing the way we name regions. If we were using scans, prewarming client cache would be near costless (as opposed to what we'll currently have to do which is first a getClosestRowBefore and then a scan from the closestrowbefore forward). Converting to the new method, we'd have to run a migration on startup changing the content in meta. Up to this, the randomid component of a region name has been the timestamp of region creation. HBASE-2531 32-bit encoding of regionnames waaay too susceptible to hash clashes proposes changing the randomid so that it contains actual name of the directory in the filesystem that hosts the region. If we had this in place, I think it would help with the migration to this new way of doing the meta because as is, the region name in fs is a hash of regionname... changing the format of the regionname would mean we generate a different hash... so we'd need hbase-2531 to be in place before we could do this change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5217) Reenable the thrift tests, and add a new one for getRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188643#comment-13188643 ] jirapos...@reviews.apache.org commented on HBASE-5217: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3514/ --- (Updated 2012-01-18 19:20:58.494454) Review request for hbase. Summary --- At some point we disabled tests for the thrift server. In addition, it looks like the getRegionInfo no longer functions. I'd like to reenable the tests and add one for getRegionInfo. This addresses bug HBASE-5217. https://issues.apache.org/jira/browse/HBASE-5217 Diffs (updated) - src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java c3be6e3 Diff: https://reviews.apache.org/r/3514/diff Testing --- Ran the tests with my changes in HBASE-2600 to test. Thanks, Alex Reenable the thrift tests, and add a new one for getRegionInfo -- Key: HBASE-5217 URL: https://issues.apache.org/jira/browse/HBASE-5217 Project: HBase Issue Type: Improvement Reporter: Alex Newman Assignee: Alex Newman Priority: Minor Attachments: 0001-Fixing-thrift-tests-v2.patch, 0001-Fixing-thrift-tests.patch, 0002-HBASE-5217.-Reenable-the-thrift-tests-and-add-a-new-v2.patch, -hbase-posix4e #92 Console [Jenkins].pdf At some point we disabled tests for the thrift server. In addition, it looks like the getRegionInfo no longer functions. I'd like to reenable the tests and add one for getRegionInfo. I had to write this to test my changes in HBASE-2600 anyway. I figured I would break it out. We shouldn't commit it until we have fixed getting the regioninfo from the thriftserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5204) Backward compatibility fixes for 0.92
[
https://issues.apache.org/jira/browse/HBASE-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188646#comment-13188646
]
Benoit Sigoure commented on HBASE-5204:
---
Actually, because entries for {{MultiPut}} and {{MultiResponse}} have been
removed, it would be better to insert {{BitComparator}} in between {{Delete
[]}} and {{HLog.Entry}}, and then put a placeholder next to it.
BTW, the follow one-liner is useful to see what codes are assigned to what
class, and diff the output between versions of HBase:
{code}
awk '/code\+\+/{code+=1; print code, $0;}'
src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java
{code}
Backward compatibility fixes for 0.92
-
Key: HBASE-5204
URL: https://issues.apache.org/jira/browse/HBASE-5204
Project: HBase
Issue Type: Bug
Components: ipc
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Blocker
Labels: backwards-compatibility
Fix For: 0.92.0
Attachments:
0001-Add-some-backward-compatible-support-for-reading-old.patch,
0002-Make-sure-that-a-connection-always-uses-a-protocol.patch,
0003-Change-the-code-used-when-serializing-HTableDescript.patch, 5204-92.txt,
5204-trunk.txt
Attached are 3 patches that are necessary to allow compatibility between
HBase 0.90.x (and previous releases) and HBase 0.92.0.
First of all, I'm well aware that 0.92.0 RC4 has been thumbed up by a lot of
people and would probably wind up being released as 0.92.0 tomorrow, so I
sincerely apologize for creating this issue so late in the process. I spent
a lot of time trying to work around the quirks of 0.92 but once I realized
that with a few very quasi-trivial changes compatibility would be made
significantly easier, I immediately sent these 3 patches to Stack, who
suggested I create this issue.
The first patch is required as without it clients sending a 0.90-style RPC to
a 0.92-style server causes the server to die uncleanly. It seems that 0.92
ships with {{\-XX:OnOutOfMemoryError=kill \-9 %p}}, and when a 0.92 server
fails to deserialize a 0.90-style RPC, it attempts to allocate a large buffer
because it doesn't read fields of 0.90-style RPCs properly. This allocation
attempt immediately triggers an OOME, which causes the JVM to die abruptly of
a {{SIGKILL}}. So whenever a 0.90.x client attempts to connect to HBase, it
kills whichever RS is hosting the {{\-ROOT-}} region.
The second patch fixes a bug introduced by HBASE-2002, which added support
for letting clients specify what protocol they want to speak. If a client
doesn't properly specify what protocol to use, the connection's {{protocol}}
field will be left {{null}}, which causes any subsequent RPC on that
connection to trigger an NPE in the server, even though the connection was
successfully established from the client's point of view. The fix is to
simply give the connection a default protocol, by assuming the client meant
to speak to a RegionServer.
The third patch fixes an oversight that slipped in HBASE-451, where a change
to {{HbaseObjectWritable}} caused all the codes used to serialize
{{Writables}} to shift by one. This was carefully avoided in other changes
such as HBASE-1502, which cleanly removed entries for {{HMsg}} and
{{HMsg[]}}, so I don't think this breakage in HBASE-451 was intended.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
[ https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188661#comment-13188661 ] Hadoop QA commented on HBASE-5179: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511020/5179-v11-92.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/807//console This message is automatically generated. Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss Key: HBASE-5179 URL: https://issues.apache.org/jira/browse/HBASE-5179 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.6 Attachments: 5179-90.txt, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch If master's processing its failover and ServerShutdownHandler's processing happen concurrently, it may appear following case. 1.master completed splitLogAfterStartup() 2.RegionserverA restarts, and ServerShutdownHandler is processing. 3.master starts to rebuildUserRegions, and RegionserverA is considered as dead server. 4.master starts to assign regions of RegionserverA because it is a dead server by step3. However, when doing step4(assigning region), ServerShutdownHandler may be doing split log, Therefore, it may cause data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5204) Backward compatibility fixes for 0.92
[
https://issues.apache.org/jira/browse/HBASE-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5204:
--
Attachment: 5204.addendum
How about this addendum ?
Backward compatibility fixes for 0.92
-
Key: HBASE-5204
URL: https://issues.apache.org/jira/browse/HBASE-5204
Project: HBase
Issue Type: Bug
Components: ipc
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Blocker
Labels: backwards-compatibility
Fix For: 0.92.0
Attachments:
0001-Add-some-backward-compatible-support-for-reading-old.patch,
0002-Make-sure-that-a-connection-always-uses-a-protocol.patch,
0003-Change-the-code-used-when-serializing-HTableDescript.patch, 5204-92.txt,
5204-trunk.txt, 5204.addendum
Attached are 3 patches that are necessary to allow compatibility between
HBase 0.90.x (and previous releases) and HBase 0.92.0.
First of all, I'm well aware that 0.92.0 RC4 has been thumbed up by a lot of
people and would probably wind up being released as 0.92.0 tomorrow, so I
sincerely apologize for creating this issue so late in the process. I spent
a lot of time trying to work around the quirks of 0.92 but once I realized
that with a few very quasi-trivial changes compatibility would be made
significantly easier, I immediately sent these 3 patches to Stack, who
suggested I create this issue.
The first patch is required as without it clients sending a 0.90-style RPC to
a 0.92-style server causes the server to die uncleanly. It seems that 0.92
ships with {{\-XX:OnOutOfMemoryError=kill \-9 %p}}, and when a 0.92 server
fails to deserialize a 0.90-style RPC, it attempts to allocate a large buffer
because it doesn't read fields of 0.90-style RPCs properly. This allocation
attempt immediately triggers an OOME, which causes the JVM to die abruptly of
a {{SIGKILL}}. So whenever a 0.90.x client attempts to connect to HBase, it
kills whichever RS is hosting the {{\-ROOT-}} region.
The second patch fixes a bug introduced by HBASE-2002, which added support
for letting clients specify what protocol they want to speak. If a client
doesn't properly specify what protocol to use, the connection's {{protocol}}
field will be left {{null}}, which causes any subsequent RPC on that
connection to trigger an NPE in the server, even though the connection was
successfully established from the client's point of view. The fix is to
simply give the connection a default protocol, by assuming the client meant
to speak to a RegionServer.
The third patch fixes an oversight that slipped in HBASE-451, where a change
to {{HbaseObjectWritable}} caused all the codes used to serialize
{{Writables}} to shift by one. This was carefully avoided in other changes
such as HBASE-1502, which cleanly removed entries for {{HMsg}} and
{{HMsg[]}}, so I don't think this breakage in HBASE-451 was intended.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5217) Reenable the thrift tests, and add a new one for getRegionInfo
[ https://issues.apache.org/jira/browse/HBASE-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188664#comment-13188664 ] Hadoop QA commented on HBASE-5217: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511026/0002-HBASE-5217.-Reenable-the-thrift-tests-and-add-a-new-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -145 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 82 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/806//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/806//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/806//console This message is automatically generated. Reenable the thrift tests, and add a new one for getRegionInfo -- Key: HBASE-5217 URL: https://issues.apache.org/jira/browse/HBASE-5217 Project: HBase Issue Type: Improvement Reporter: Alex Newman Assignee: Alex Newman Priority: Minor Attachments: 0001-Fixing-thrift-tests-v2.patch, 0001-Fixing-thrift-tests.patch, 0002-HBASE-5217.-Reenable-the-thrift-tests-and-add-a-new-v2.patch, -hbase-posix4e #92 Console [Jenkins].pdf At some point we disabled tests for the thrift server. In addition, it looks like the getRegionInfo no longer functions. I'd like to reenable the tests and add one for getRegionInfo. I had to write this to test my changes in HBASE-2600 anyway. I figured I would break it out. We shouldn't commit it until we have fixed getting the regioninfo from the thriftserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
[ https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5179: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511020/5179-v11-92.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/807//console This message is automatically generated.) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss Key: HBASE-5179 URL: https://issues.apache.org/jira/browse/HBASE-5179 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.6 Attachments: 5179-90.txt, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch If master's processing its failover and ServerShutdownHandler's processing happen concurrently, it may appear following case. 1.master completed splitLogAfterStartup() 2.RegionserverA restarts, and ServerShutdownHandler is processing. 3.master starts to rebuildUserRegions, and RegionserverA is considered as dead server. 4.master starts to assign regions of RegionserverA because it is a dead server by step3. However, when doing step4(assigning region), ServerShutdownHandler may be doing split log, Therefore, it may cause data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2600) Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid
[ https://issues.apache.org/jira/browse/HBASE-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188679#comment-13188679 ] Hadoop QA commented on HBASE-2600: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511025/0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v9.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 31 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -145 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 81 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestImportTsv Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/808//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/808//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/808//console This message is automatically generated. Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid Key: HBASE-2600 URL: https://issues.apache.org/jira/browse/HBASE-2600 Project: HBase Issue Type: Bug Reporter: stack Assignee: Alex Newman Attachments: 0001-Changed-regioninfo-format-to-use-endKey-instead-of-s.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v2.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v4.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v6.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v7.2.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8.1, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v9.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen.patch, 2600-trunk-01-17.txt, jenkins.pdf This is an idea that Ryan and I have been kicking around on and off for a while now. If regionnames were made of tablename+endrow instead of tablename+startrow, then in the metatables, doing a search for the region that contains the wanted row, we'd just have to open a scanner using passed row and the first row found by the scan would be that of the region we need (If offlined parent, we'd have to scan to the next row). If we redid the meta tables in this format, we'd be using an access that is natural to hbase, a scan as opposed to the perverse, expensive getClosestRowBefore we currently have that has to walk backward in meta finding a containing region. This issue is about changing the way we name regions. If we were using scans, prewarming client cache would be near costless (as opposed to what we'll currently have to do which is first a getClosestRowBefore and then a scan from the closestrowbefore forward). Converting to the new method, we'd have to run a migration on startup changing the content in meta. Up to this, the randomid component of a region name has been the timestamp of region creation. HBASE-2531 32-bit encoding of regionnames waaay too susceptible to hash clashes proposes changing the randomid so that it contains actual name of the directory in the filesystem that hosts the region. If we had this in place, I think it would help with the migration to this new way of doing the meta because as is, the region name in fs is a hash of regionname... changing the format of the regionname would mean we generate a different hash... so we'd need hbase-2531 to be in place before we could do this change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files before compaction selection
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188681#comment-13188681 ] Phabricator commented on HBASE-5199: tedyu has commented on the revision [jira][HBASE-5199] Delete out of TTL store files before compaction selection . INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1053 Should this check be moved to line 1016 ? src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1024 Changing 'because' to 'where' would make the sentence more readable REVISION DETAIL https://reviews.facebook.net/D1311 Delete out of TTL store files before compaction selection - Key: HBASE-5199 URL: https://issues.apache.org/jira/browse/HBASE-5199 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D1311.1.patch Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions. In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
[ https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188686#comment-13188686 ] Hadoop QA commented on HBASE-5179: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511030/5179-90v10.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/809//console This message is automatically generated. Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss Key: HBASE-5179 URL: https://issues.apache.org/jira/browse/HBASE-5179 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.6 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch If master's processing its failover and ServerShutdownHandler's processing happen concurrently, it may appear following case. 1.master completed splitLogAfterStartup() 2.RegionserverA restarts, and ServerShutdownHandler is processing. 3.master starts to rebuildUserRegions, and RegionserverA is considered as dead server. 4.master starts to assign regions of RegionserverA because it is a dead server by step3. However, when doing step4(assigning region), ServerShutdownHandler may be doing split log, Therefore, it may cause data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
[ https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5179: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511030/5179-90v10.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/809//console This message is automatically generated.) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss Key: HBASE-5179 URL: https://issues.apache.org/jira/browse/HBASE-5179 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.6 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch If master's processing its failover and ServerShutdownHandler's processing happen concurrently, it may appear following case. 1.master completed splitLogAfterStartup() 2.RegionserverA restarts, and ServerShutdownHandler is processing. 3.master starts to rebuildUserRegions, and RegionserverA is considered as dead server. 4.master starts to assign regions of RegionserverA because it is a dead server by step3. However, when doing step4(assigning region), ServerShutdownHandler may be doing split log, Therefore, it may cause data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5204) Backward compatibility fixes for 0.92
[
https://issues.apache.org/jira/browse/HBASE-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5204:
--
Attachment: (was: 5204.addendum)
Backward compatibility fixes for 0.92
-
Key: HBASE-5204
URL: https://issues.apache.org/jira/browse/HBASE-5204
Project: HBase
Issue Type: Bug
Components: ipc
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Blocker
Labels: backwards-compatibility
Fix For: 0.92.0
Attachments:
0001-Add-some-backward-compatible-support-for-reading-old.patch,
0002-Make-sure-that-a-connection-always-uses-a-protocol.patch,
0003-Change-the-code-used-when-serializing-HTableDescript.patch, 5204-92.txt,
5204-trunk.txt, 5204.addendum
Attached are 3 patches that are necessary to allow compatibility between
HBase 0.90.x (and previous releases) and HBase 0.92.0.
First of all, I'm well aware that 0.92.0 RC4 has been thumbed up by a lot of
people and would probably wind up being released as 0.92.0 tomorrow, so I
sincerely apologize for creating this issue so late in the process. I spent
a lot of time trying to work around the quirks of 0.92 but once I realized
that with a few very quasi-trivial changes compatibility would be made
significantly easier, I immediately sent these 3 patches to Stack, who
suggested I create this issue.
The first patch is required as without it clients sending a 0.90-style RPC to
a 0.92-style server causes the server to die uncleanly. It seems that 0.92
ships with {{\-XX:OnOutOfMemoryError=kill \-9 %p}}, and when a 0.92 server
fails to deserialize a 0.90-style RPC, it attempts to allocate a large buffer
because it doesn't read fields of 0.90-style RPCs properly. This allocation
attempt immediately triggers an OOME, which causes the JVM to die abruptly of
a {{SIGKILL}}. So whenever a 0.90.x client attempts to connect to HBase, it
kills whichever RS is hosting the {{\-ROOT-}} region.
The second patch fixes a bug introduced by HBASE-2002, which added support
for letting clients specify what protocol they want to speak. If a client
doesn't properly specify what protocol to use, the connection's {{protocol}}
field will be left {{null}}, which causes any subsequent RPC on that
connection to trigger an NPE in the server, even though the connection was
successfully established from the client's point of view. The fix is to
simply give the connection a default protocol, by assuming the client meant
to speak to a RegionServer.
The third patch fixes an oversight that slipped in HBASE-451, where a change
to {{HbaseObjectWritable}} caused all the codes used to serialize
{{Writables}} to shift by one. This was carefully avoided in other changes
such as HBASE-1502, which cleanly removed entries for {{HMsg}} and
{{HMsg[]}}, so I don't think this breakage in HBASE-451 was intended.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5204) Backward compatibility fixes for 0.92
[
https://issues.apache.org/jira/browse/HBASE-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5204:
--
Attachment: 5204.addendum
This is the correct addendum.
Backward compatibility fixes for 0.92
-
Key: HBASE-5204
URL: https://issues.apache.org/jira/browse/HBASE-5204
Project: HBase
Issue Type: Bug
Components: ipc
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Blocker
Labels: backwards-compatibility
Fix For: 0.92.0
Attachments:
0001-Add-some-backward-compatible-support-for-reading-old.patch,
0002-Make-sure-that-a-connection-always-uses-a-protocol.patch,
0003-Change-the-code-used-when-serializing-HTableDescript.patch, 5204-92.txt,
5204-trunk.txt, 5204.addendum
Attached are 3 patches that are necessary to allow compatibility between
HBase 0.90.x (and previous releases) and HBase 0.92.0.
First of all, I'm well aware that 0.92.0 RC4 has been thumbed up by a lot of
people and would probably wind up being released as 0.92.0 tomorrow, so I
sincerely apologize for creating this issue so late in the process. I spent
a lot of time trying to work around the quirks of 0.92 but once I realized
that with a few very quasi-trivial changes compatibility would be made
significantly easier, I immediately sent these 3 patches to Stack, who
suggested I create this issue.
The first patch is required as without it clients sending a 0.90-style RPC to
a 0.92-style server causes the server to die uncleanly. It seems that 0.92
ships with {{\-XX:OnOutOfMemoryError=kill \-9 %p}}, and when a 0.92 server
fails to deserialize a 0.90-style RPC, it attempts to allocate a large buffer
because it doesn't read fields of 0.90-style RPCs properly. This allocation
attempt immediately triggers an OOME, which causes the JVM to die abruptly of
a {{SIGKILL}}. So whenever a 0.90.x client attempts to connect to HBase, it
kills whichever RS is hosting the {{\-ROOT-}} region.
The second patch fixes a bug introduced by HBASE-2002, which added support
for letting clients specify what protocol they want to speak. If a client
doesn't properly specify what protocol to use, the connection's {{protocol}}
field will be left {{null}}, which causes any subsequent RPC on that
connection to trigger an NPE in the server, even though the connection was
successfully established from the client's point of view. The fix is to
simply give the connection a default protocol, by assuming the client meant
to speak to a RegionServer.
The third patch fixes an oversight that slipped in HBASE-451, where a change
to {{HbaseObjectWritable}} caused all the codes used to serialize
{{Writables}} to shift by one. This was carefully avoided in other changes
such as HBASE-1502, which cleanly removed entries for {{HMsg}} and
{{HMsg[]}}, so I don't think this breakage in HBASE-451 was intended.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5204) Backward compatibility fixes for 0.92
[
https://issues.apache.org/jira/browse/HBASE-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188691#comment-13188691
]
Zhihong Yu commented on HBASE-5204:
---
Here is the diff after addendum is applied:
{code}
zhihyu$ diff ../90hbase/obj-writable.90 obj-writable.92
20,21c20,21
20 addToMap(HMsg.class, code++);
21 addToMap(HMsg[].class, code++);
---
20 addToMap(Integer.class, code++);
21 addToMap(Integer[].class, code++);
57,58c57,58
57 addToMap(MultiPut.class, code++);
58 addToMap(MultiPutResponse.class, code++);
---
57 addToMap(BitComparator.class, code++);
58 addToMap(Exec.class, code++);
71a72,77
72 addToMap(RandomRowFilter.class, code++);
73 addToMap(CompareOp.class, code++);
74 addToMap(ColumnRangeFilter.class, code++);
75 addToMap(HServerLoad.class, code++);
76 addToMap(RegionOpeningState.class, code++);
77 addToMap(HTableDescriptor[].class, code++);
{code}
Backward compatibility fixes for 0.92
-
Key: HBASE-5204
URL: https://issues.apache.org/jira/browse/HBASE-5204
Project: HBase
Issue Type: Bug
Components: ipc
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Blocker
Labels: backwards-compatibility
Fix For: 0.92.0
Attachments:
0001-Add-some-backward-compatible-support-for-reading-old.patch,
0002-Make-sure-that-a-connection-always-uses-a-protocol.patch,
0003-Change-the-code-used-when-serializing-HTableDescript.patch, 5204-92.txt,
5204-trunk.txt, 5204.addendum
Attached are 3 patches that are necessary to allow compatibility between
HBase 0.90.x (and previous releases) and HBase 0.92.0.
First of all, I'm well aware that 0.92.0 RC4 has been thumbed up by a lot of
people and would probably wind up being released as 0.92.0 tomorrow, so I
sincerely apologize for creating this issue so late in the process. I spent
a lot of time trying to work around the quirks of 0.92 but once I realized
that with a few very quasi-trivial changes compatibility would be made
significantly easier, I immediately sent these 3 patches to Stack, who
suggested I create this issue.
The first patch is required as without it clients sending a 0.90-style RPC to
a 0.92-style server causes the server to die uncleanly. It seems that 0.92
ships with {{\-XX:OnOutOfMemoryError=kill \-9 %p}}, and when a 0.92 server
fails to deserialize a 0.90-style RPC, it attempts to allocate a large buffer
because it doesn't read fields of 0.90-style RPCs properly. This allocation
attempt immediately triggers an OOME, which causes the JVM to die abruptly of
a {{SIGKILL}}. So whenever a 0.90.x client attempts to connect to HBase, it
kills whichever RS is hosting the {{\-ROOT-}} region.
The second patch fixes a bug introduced by HBASE-2002, which added support
for letting clients specify what protocol they want to speak. If a client
doesn't properly specify what protocol to use, the connection's {{protocol}}
field will be left {{null}}, which causes any subsequent RPC on that
connection to trigger an NPE in the server, even though the connection was
successfully established from the client's point of view. The fix is to
simply give the connection a default protocol, by assuming the client meant
to speak to a RegionServer.
The third patch fixes an oversight that slipped in HBASE-451, where a change
to {{HbaseObjectWritable}} caused all the codes used to serialize
{{Writables}} to shift by one. This was carefully avoided in other changes
such as HBASE-1502, which cleanly removed entries for {{HMsg}} and
{{HMsg[]}}, so I don't think this breakage in HBASE-451 was intended.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5204) Backward compatibility fixes for 0.92
[
https://issues.apache.org/jira/browse/HBASE-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188699#comment-13188699
]
Zhihong Yu commented on HBASE-5204:
---
Will integrate addendum in one hour if there is no objection.
Backward compatibility fixes for 0.92
-
Key: HBASE-5204
URL: https://issues.apache.org/jira/browse/HBASE-5204
Project: HBase
Issue Type: Bug
Components: ipc
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Blocker
Labels: backwards-compatibility
Fix For: 0.92.0
Attachments:
0001-Add-some-backward-compatible-support-for-reading-old.patch,
0002-Make-sure-that-a-connection-always-uses-a-protocol.patch,
0003-Change-the-code-used-when-serializing-HTableDescript.patch, 5204-92.txt,
5204-trunk.txt, 5204.addendum
Attached are 3 patches that are necessary to allow compatibility between
HBase 0.90.x (and previous releases) and HBase 0.92.0.
First of all, I'm well aware that 0.92.0 RC4 has been thumbed up by a lot of
people and would probably wind up being released as 0.92.0 tomorrow, so I
sincerely apologize for creating this issue so late in the process. I spent
a lot of time trying to work around the quirks of 0.92 but once I realized
that with a few very quasi-trivial changes compatibility would be made
significantly easier, I immediately sent these 3 patches to Stack, who
suggested I create this issue.
The first patch is required as without it clients sending a 0.90-style RPC to
a 0.92-style server causes the server to die uncleanly. It seems that 0.92
ships with {{\-XX:OnOutOfMemoryError=kill \-9 %p}}, and when a 0.92 server
fails to deserialize a 0.90-style RPC, it attempts to allocate a large buffer
because it doesn't read fields of 0.90-style RPCs properly. This allocation
attempt immediately triggers an OOME, which causes the JVM to die abruptly of
a {{SIGKILL}}. So whenever a 0.90.x client attempts to connect to HBase, it
kills whichever RS is hosting the {{\-ROOT-}} region.
The second patch fixes a bug introduced by HBASE-2002, which added support
for letting clients specify what protocol they want to speak. If a client
doesn't properly specify what protocol to use, the connection's {{protocol}}
field will be left {{null}}, which causes any subsequent RPC on that
connection to trigger an NPE in the server, even though the connection was
successfully established from the client's point of view. The fix is to
simply give the connection a default protocol, by assuming the client meant
to speak to a RegionServer.
The third patch fixes an oversight that slipped in HBASE-451, where a change
to {{HbaseObjectWritable}} caused all the codes used to serialize
{{Writables}} to shift by one. This was carefully avoided in other changes
such as HBASE-1502, which cleanly removed entries for {{HMsg}} and
{{HMsg[]}}, so I don't think this breakage in HBASE-451 was intended.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3584) Allow atomic put/delete in one call
[ https://issues.apache.org/jira/browse/HBASE-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188705#comment-13188705 ] jirapos...@reviews.apache.org commented on HBASE-3584: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3481/#review4445 --- http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/3481/#comment9976 @Lars: Sorry about the late question. I would like to understand the following case. What if we crash in between this loop? Is it possible that we have persisted half the WALEdits, but not gotten to the rest? I am concerned, that if that were the case (hopefully not) then on RS restart, we will replay, a partial list of mutations from arm. - Amitanand On 2012-01-13 17:11:21, Lars Hofhansl wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/3481/ bq. --- bq. bq. (Updated 2012-01-13 17:11:21) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Add API for atomic row mutations to HBase (currently Put and Delete). bq. bq. Client API would look like this: bq. Delete d = new Delete(ROW); bq. Put p = new Put(ROW); bq. ... bq. AtomicRowMutation arm = new AtomicRowMutation(ROW); bq. arm.add(p); bq. arm.add(d); bq. myHtable.mutateAtomically(arm); bq. bq. bq. This addresses bug HBASE-3584. bq. https://issues.apache.org/jira/browse/HBASE-3584 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AtomicRowMutation.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java 1231172 bq. bq. Diff: https://reviews.apache.org/r/3481/diff bq. bq. bq. Testing bq. --- bq. bq. * Simple functional test: TestFromClientSide.testAtomicRowMutation bq. * Multithreaded stress test: TestAtomicOperation.testAtomicMutationMultiThreads bq. * coprocessor test: TestRegionObserverInterface.testAtomicRowMutation bq. * manual testing bq. bq. bq. Thanks, bq. bq. Lars bq. bq. Allow atomic put/delete in one call --- Key: HBASE-3584 URL: https://issues.apache.org/jira/browse/HBASE-3584 Project: HBase Issue Type: New Feature Components: client, coprocessors, regionserver Reporter: ryan rawson Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 3584-final.txt, 3584-v1.txt, 3584-v3.txt Right now we have the following calls: put(Put) delete(Delete) increment(Increments) But we cannot combine all of the above in a single call, complete with a single row lock. It would be nice to do that. It would also allow us to do a CAS where we could do a put/increment if the check succeeded. - Amendment: Since Increment does not currently support MVCC it cannot be included in an atomic operation. So this for Put and Delete only. -- This message is automatically generated by JIRA. If you think it was sent incorrectly,
[jira] [Commented] (HBASE-5204) Backward compatibility fixes for 0.92
[
https://issues.apache.org/jira/browse/HBASE-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188711#comment-13188711
]
stack commented on HBASE-5204:
--
Zhihong Would suggest you wait on Benoît feedback.
This means a new RC? Thats kinda obscene. In your client B, can you not
rejigger the codes dependent on hbase version?
Backward compatibility fixes for 0.92
-
Key: HBASE-5204
URL: https://issues.apache.org/jira/browse/HBASE-5204
Project: HBase
Issue Type: Bug
Components: ipc
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Blocker
Labels: backwards-compatibility
Fix For: 0.92.0
Attachments:
0001-Add-some-backward-compatible-support-for-reading-old.patch,
0002-Make-sure-that-a-connection-always-uses-a-protocol.patch,
0003-Change-the-code-used-when-serializing-HTableDescript.patch, 5204-92.txt,
5204-trunk.txt, 5204.addendum
Attached are 3 patches that are necessary to allow compatibility between
HBase 0.90.x (and previous releases) and HBase 0.92.0.
First of all, I'm well aware that 0.92.0 RC4 has been thumbed up by a lot of
people and would probably wind up being released as 0.92.0 tomorrow, so I
sincerely apologize for creating this issue so late in the process. I spent
a lot of time trying to work around the quirks of 0.92 but once I realized
that with a few very quasi-trivial changes compatibility would be made
significantly easier, I immediately sent these 3 patches to Stack, who
suggested I create this issue.
The first patch is required as without it clients sending a 0.90-style RPC to
a 0.92-style server causes the server to die uncleanly. It seems that 0.92
ships with {{\-XX:OnOutOfMemoryError=kill \-9 %p}}, and when a 0.92 server
fails to deserialize a 0.90-style RPC, it attempts to allocate a large buffer
because it doesn't read fields of 0.90-style RPCs properly. This allocation
attempt immediately triggers an OOME, which causes the JVM to die abruptly of
a {{SIGKILL}}. So whenever a 0.90.x client attempts to connect to HBase, it
kills whichever RS is hosting the {{\-ROOT-}} region.
The second patch fixes a bug introduced by HBASE-2002, which added support
for letting clients specify what protocol they want to speak. If a client
doesn't properly specify what protocol to use, the connection's {{protocol}}
field will be left {{null}}, which causes any subsequent RPC on that
connection to trigger an NPE in the server, even though the connection was
successfully established from the client's point of view. The fix is to
simply give the connection a default protocol, by assuming the client meant
to speak to a RegionServer.
The third patch fixes an oversight that slipped in HBASE-451, where a change
to {{HbaseObjectWritable}} caused all the codes used to serialize
{{Writables}} to shift by one. This was carefully avoided in other changes
such as HBASE-1502, which cleanly removed entries for {{HMsg}} and
{{HMsg[]}}, so I don't think this breakage in HBASE-451 was intended.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5208) Allow setting Scan start/stop row individually in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188709#comment-13188709 ] Zhihong Yu commented on HBASE-5208: --- Integrated to TRUNK. Thanks for the patch Nicolas. Allow setting Scan start/stop row individually in TableInputFormat -- Key: HBASE-5208 URL: https://issues.apache.org/jira/browse/HBASE-5208 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Nicholas Telford Priority: Minor Attachments: HBASE-5208-001.txt, HBASE-5208-002.txt, HBASE-5208-003.txt, HBASE-5208-004.txt Currently, TableInputFormat initializes a serialized Scan from hbase.mapreduce.scan. Alternatively, it will instantiate a new Scan using properties defined in hbase.mapreduce.scan.*. However, of these properties the start row and stop row (arguably the most pertinent) are missing. TableInputFormat should permit the specification of a start/stop row as with the other fields using a new pair of properties: hbase.mapreduce.scan.row.start and hbase.mapreduce.scan.row.end The primary use-case for this is to permit Oozie and other job management tools that can't call TableMapReduceUtil.initTableMapperJob() to operate on a contiguous subset of rows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files before compaction selection
[
https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188715#comment-13188715
]
Phabricator commented on HBASE-5199:
Karthik has commented on the revision [jira][HBASE-5199] Delete out of TTL
store files before compaction selection .
Havent reviewed the test code yet...
INLINE COMMENTS
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1039 Do we need
this assignment? Can we make the condition:
if (nonExpiredFiles.size() currentStoreFiles.size()) {
//...
}
src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1017 Should we
exclude files contained in filesCompacting? Otherwise we could end up deleting
a file which is currently being compacted...
REVISION DETAIL
https://reviews.facebook.net/D1311
Delete out of TTL store files before compaction selection
-
Key: HBASE-5199
URL: https://issues.apache.org/jira/browse/HBASE-5199
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
Attachments: D1311.1.patch
Currently, HBase deletes the out of TTL store files after compaction. We can
change the sequence to delete the out of TTL store files before selecting
store files for compactions.
In this way, HBase can keep deleting the old invalid store files without
compaction, and also prevent from unnecessary compactions since the out of
TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
[ https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5179: -- Attachment: 5179-90v11.patch Patch v11 for 0.90 branch. Applied same simplification technique to DeadServer.java Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss Key: HBASE-5179 URL: https://issues.apache.org/jira/browse/HBASE-5179 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.6 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch If master's processing its failover and ServerShutdownHandler's processing happen concurrently, it may appear following case. 1.master completed splitLogAfterStartup() 2.RegionserverA restarts, and ServerShutdownHandler is processing. 3.master starts to rebuildUserRegions, and RegionserverA is considered as dead server. 4.master starts to assign regions of RegionserverA because it is a dead server by step3. However, when doing step4(assigning region), ServerShutdownHandler may be doing split log, Therefore, it may cause data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
[ https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188737#comment-13188737 ] Zhihong Yu commented on HBASE-5179: --- @Stack: Do you want to take a look at the v11 patches (3 of them) ? Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss Key: HBASE-5179 URL: https://issues.apache.org/jira/browse/HBASE-5179 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.6 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch If master's processing its failover and ServerShutdownHandler's processing happen concurrently, it may appear following case. 1.master completed splitLogAfterStartup() 2.RegionserverA restarts, and ServerShutdownHandler is processing. 3.master starts to rebuildUserRegions, and RegionserverA is considered as dead server. 4.master starts to assign regions of RegionserverA because it is a dead server by step3. However, when doing step4(assigning region), ServerShutdownHandler may be doing split log, Therefore, it may cause data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before compaction selection
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5199: --- Attachment: D1311.2.patch Liyin updated the revision [jira][HBASE-5199] Delete out of TTL store files before compaction selection . Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik Address Karthik's comments. REVISION DETAIL https://reviews.facebook.net/D1311 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/regionserver/Store.java src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java Delete out of TTL store files before compaction selection - Key: HBASE-5199 URL: https://issues.apache.org/jira/browse/HBASE-5199 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D1311.1.patch, D1311.2.patch Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions. In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3584) Allow atomic put/delete in one call
[ https://issues.apache.org/jira/browse/HBASE-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188740#comment-13188740 ] jirapos...@reviews.apache.org commented on HBASE-3584: -- bq. On 2012-01-18 21:18:16, Amitanand Aiyer wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 4187 bq. https://reviews.apache.org/r/3481/diff/4/?file=68814#file68814line4187 bq. bq. @Lars: Sorry about the late question. I would like to understand the following case. bq. bq. What if we crash in between this loop? bq. bq. Is it possible that we have persisted half the WALEdits, but not gotten to the rest? bq. bq. I am concerned, that if that were the case (hopefully not) then on RS restart, we will replay, a partial list of mutations from arm. bq. bq. bq. bq. Yes, which is why I filed, implemented, and committed HBASE-5203, which fixes the problem for real :) - Lars --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3481/#review4445 --- On 2012-01-13 17:11:21, Lars Hofhansl wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/3481/ bq. --- bq. bq. (Updated 2012-01-13 17:11:21) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Add API for atomic row mutations to HBase (currently Put and Delete). bq. bq. Client API would look like this: bq. Delete d = new Delete(ROW); bq. Put p = new Put(ROW); bq. ... bq. AtomicRowMutation arm = new AtomicRowMutation(ROW); bq. arm.add(p); bq. arm.add(d); bq. myHtable.mutateAtomically(arm); bq. bq. bq. This addresses bug HBASE-3584. bq. https://issues.apache.org/jira/browse/HBASE-3584 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/AtomicRowMutation.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java 1231172 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java 1231172 bq. bq. Diff: https://reviews.apache.org/r/3481/diff bq. bq. bq. Testing bq. --- bq. bq. * Simple functional test: TestFromClientSide.testAtomicRowMutation bq. * Multithreaded stress test: TestAtomicOperation.testAtomicMutationMultiThreads bq. * coprocessor test: TestRegionObserverInterface.testAtomicRowMutation bq. * manual testing bq. bq. bq. Thanks, bq. bq. Lars bq. bq. Allow atomic put/delete in one call --- Key: HBASE-3584 URL: https://issues.apache.org/jira/browse/HBASE-3584 Project: HBase Issue Type: New Feature Components: client, coprocessors, regionserver Reporter: ryan rawson Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 3584-final.txt, 3584-v1.txt, 3584-v3.txt Right now we have the following calls: put(Put) delete(Delete) increment(Increments) But we cannot combine all of the above in a single call, complete with a single row lock. It would be nice to do that. It would also allow us to do a CAS where we could do a put/increment if
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188741#comment-13188741 ] Phabricator commented on HBASE-4218: Kannan has commented on the revision [jira] [HBASE-4218] HFile data block encoding framework and delta encoding implementation. another partial round of comments. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/deltaencoder/PrefixKeyDeltaEncoder.java:45 rename offset to prevKeyOffset for clarity. src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncoderAlgorithms.java:115 has - have src/main/java/org/apache/hadoop/hbase/io/deltaencoder/DeltaEncoderAlgorithms.java:124 DeltaEncoder - DeltaEncoders src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:93 Another related question/clarification needed on the spec of this feature... If delta-encoding is on in cache, then is blocksize setting for the CF based on the encoded size or the un-encoded size. [Personally, think the encoded size should be used for the blocksize. But can you clarify either way.] src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java:196 getDeltaEncodingId() and getDeltaEncodedId() seem to be identical, but for their names. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockDeltaEncoder.java:78 operate - operates src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockDeltaEncoder.java:112 I wasn't clear what useEncodedScanner is meant for. Currently, it seems to be used HFileReaderV1. Could you clarify the purpose of this... src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java:98 remove in cache src/main/java/org/apache/hadoop/hbase/KeyValue.java:1901 sounds like b.length should be l on this line also. Is this is a pre-existing bug? src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheKey.java:84 Why 2 * ClassSize.REFERENCE? This change adds one reference to the encoding enum, correct? src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java:752 We can use a MutableInteger here to avoid creating lots of Integer objects. But, since this is just for a test, not a big deal. src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java:30 KeyValue - KeyValues src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java:32 iterated always - always iterated src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java:79 Must the implementation make a deep copy? Or is it legal for the implementation to point to have the returned ByteBuffer point to a byte array in the input block? src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java:129 confusing comment. same key as what? Should comment be something like: Seek to specified key in the block. ? src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java:137 blockSeekTo -- seekToKeyInBlock, perhaps? src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java:87 this logic appears to assume that the target (this) we are copying into was positioned at the previous key. No? REVISION DETAIL https://reviews.facebook.net/D447 Data Block Encoding of KeyValues (aka delta encoding / prefix compression) --- Key: HBASE-4218 URL: https://issues.apache.org/jira/browse/HBASE-4218 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.94.0 Reporter: Jacek Migdal Assignee: Mikhail Bautin Labels: compression Fix For: 0.94.0 Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, 0001-Delta-encoding.patch, 4218-2012-01-14.txt, 4218-v16.txt, 4218.txt, D447.1.patch, D447.10.patch, D447.11.patch, D447.12.patch, D447.13.patch, D447.14.patch, D447.15.patch, D447.16.patch, D447.17.patch, D447.18.patch, D447.19.patch, D447.2.patch, D447.20.patch, D447.21.patch, D447.22.patch, D447.23.patch, D447.24.patch, D447.3.patch, D447.4.patch, D447.5.patch, D447.6.patch, D447.7.patch, D447.8.patch, D447.9.patch, Data-block-encoding-2011-12-23.patch, Delta-encoding-2012-01-17_11_09_09.patch, Delta-encoding.patch-2011-12-22_11_52_07.patch, Delta-encoding.patch-2012-01-05_15_16_43.patch, Delta-encoding.patch-2012-01-05_16_31_44.patch, Delta-encoding.patch-2012-01-05_16_31_44_copy.patch, Delta-encoding.patch-2012-01-05_18_50_47.patch, Delta-encoding.patch-2012-01-07_14_12_48.patch, Delta-encoding.patch-2012-01-13_12_20_07.patch, Delta_encoding_with_memstore_TS.patch, open-source.diff A compression for keys. Keys are sorted in HFile and they are usually very similar. Because of that, it is possible to design better compression than general purpose algorithms, It is an
[jira] [Updated] (HBASE-5199) Delete out of TTL store files before compaction selection
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5199: --- Attachment: D1311.3.patch Liyin updated the revision [jira][HBASE-5199] Delete out of TTL store files before compaction selection . Reviewers: Kannan, JIRA, khemani, aaiyer, Karthik Fix a typo. REVISION DETAIL https://reviews.facebook.net/D1311 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/regionserver/Store.java src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java Delete out of TTL store files before compaction selection - Key: HBASE-5199 URL: https://issues.apache.org/jira/browse/HBASE-5199 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions. In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5228) [REST] Rip out transform feature
[REST] Rip out transform feature -- Key: HBASE-5228 URL: https://issues.apache.org/jira/browse/HBASE-5228 Project: HBase Issue Type: Bug Affects Versions: 0.90.5, 0.92.0, 0.94.0 Reporter: Andrew Purtell Assignee: Andrew Purtell The 'transform' feature, where REST can be instructed, via a table attribute, to apply a transformation (e.g. base64 encoding or decoding) to a (sub)set of column values before serving them up to a client or storing them into HBase, was added some time ago at the request of Jack Levin. I have since come to regret it, it was not a well thought out feature: - This is really an application concern. - It adds significant overhead to request processing: Periodically a HBaseAdmin is used to retrieve the table descriptor, in order to scan through table attributes for transformation directives. I think it is best to rip it out, its a real problem area, and REST should be no more concerned about data formats than the Java API. I doubt anyone uses this, not even Jack. Will need to follow up with him to confirm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files before compaction selection
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188748#comment-13188748 ] Phabricator commented on HBASE-5199: Karthik has accepted the revision [jira][HBASE-5199] Delete out of TTL store files before compaction selection . LGTM! REVISION DETAIL https://reviews.facebook.net/D1311 Delete out of TTL store files before compaction selection - Key: HBASE-5199 URL: https://issues.apache.org/jira/browse/HBASE-5199 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions. In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5228) [REST] Rip out transform feature
[ https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-5228: -- Attachment: HBASE-5228.patch [REST] Rip out transform feature -- Key: HBASE-5228 URL: https://issues.apache.org/jira/browse/HBASE-5228 Project: HBase Issue Type: Bug Affects Versions: 0.92.0, 0.94.0, 0.90.5 Reporter: Andrew Purtell Assignee: Andrew Purtell Attachments: HBASE-5228.patch The 'transform' feature, where REST can be instructed, via a table attribute, to apply a transformation (e.g. base64 encoding or decoding) to a (sub)set of column values before serving them up to a client or storing them into HBase, was added some time ago at the request of Jack Levin. I have since come to regret it, it was not a well thought out feature: - This is really an application concern. - It adds significant overhead to request processing: Periodically a HBaseAdmin is used to retrieve the table descriptor, in order to scan through table attributes for transformation directives. I think it is best to rip it out, its a real problem area, and REST should be no more concerned about data formats than the Java API. I doubt anyone uses this, not even Jack. Will need to follow up with him to confirm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4218) Data Block Encoding of KeyValues (aka delta encoding / prefix compression)
[ https://issues.apache.org/jira/browse/HBASE-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188767#comment-13188767 ] Phabricator commented on HBASE-4218: tedyu has commented on the revision [jira] [HBASE-4218] HFile data block encoding framework and delta encoding implementation. INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java:130 Should read 'array where the key is' REVISION DETAIL https://reviews.facebook.net/D447 Data Block Encoding of KeyValues (aka delta encoding / prefix compression) --- Key: HBASE-4218 URL: https://issues.apache.org/jira/browse/HBASE-4218 Project: HBase Issue Type: Improvement Components: io Affects Versions: 0.94.0 Reporter: Jacek Migdal Assignee: Mikhail Bautin Labels: compression Fix For: 0.94.0 Attachments: 0001-Delta-encoding-fixed-encoded-scanners.patch, 0001-Delta-encoding.patch, 4218-2012-01-14.txt, 4218-v16.txt, 4218.txt, D447.1.patch, D447.10.patch, D447.11.patch, D447.12.patch, D447.13.patch, D447.14.patch, D447.15.patch, D447.16.patch, D447.17.patch, D447.18.patch, D447.19.patch, D447.2.patch, D447.20.patch, D447.21.patch, D447.22.patch, D447.23.patch, D447.24.patch, D447.3.patch, D447.4.patch, D447.5.patch, D447.6.patch, D447.7.patch, D447.8.patch, D447.9.patch, Data-block-encoding-2011-12-23.patch, Delta-encoding-2012-01-17_11_09_09.patch, Delta-encoding.patch-2011-12-22_11_52_07.patch, Delta-encoding.patch-2012-01-05_15_16_43.patch, Delta-encoding.patch-2012-01-05_16_31_44.patch, Delta-encoding.patch-2012-01-05_16_31_44_copy.patch, Delta-encoding.patch-2012-01-05_18_50_47.patch, Delta-encoding.patch-2012-01-07_14_12_48.patch, Delta-encoding.patch-2012-01-13_12_20_07.patch, Delta_encoding_with_memstore_TS.patch, open-source.diff A compression for keys. Keys are sorted in HFile and they are usually very similar. Because of that, it is possible to design better compression than general purpose algorithms, It is an additional step designed to be used in memory. It aims to save memory in cache as well as speeding seeks within HFileBlocks. It should improve performance a lot, if key lengths are larger than value lengths. For example, it makes a lot of sense to use it when value is a counter. Initial tests on real data (key length = ~ 90 bytes , value length = 8 bytes) shows that I could achieve decent level of compression: key compression ratio: 92% total compression ratio: 85% LZO on the same data: 85% LZO after delta encoding: 91% While having much better performance (20-80% faster decompression ratio than LZO). Moreover, it should allow far more efficient seeking which should improve performance a bit. It seems that a simple compression algorithms are good enough. Most of the savings are due to prefix compression, int128 encoding, timestamp diffs and bitfields to avoid duplication. That way, comparisons of compressed data can be much faster than a byte comparator (thanks to prefix compression and bitfields). In order to implement it in HBase two important changes in design will be needed: -solidify interface to HFileBlock / HFileReader Scanner to provide seeking and iterating; access to uncompressed buffer in HFileBlock will have bad performance -extend comparators to support comparison assuming that N first bytes are equal (or some fields are equal) Link to a discussion about something similar: http://search-hadoop.com/m/5aqGXJEnaD1/hbase+windowssubj=Re+prefix+compression -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5204) Backward compatibility fixes for 0.92
[
https://issues.apache.org/jira/browse/HBASE-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188776#comment-13188776
]
stack commented on HBASE-5204:
--
Offline I talked to Benoit. Changing order is a nice to have. This issue is
not an RC blocker.
Backward compatibility fixes for 0.92
-
Key: HBASE-5204
URL: https://issues.apache.org/jira/browse/HBASE-5204
Project: HBase
Issue Type: Bug
Components: ipc
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Blocker
Labels: backwards-compatibility
Fix For: 0.92.0
Attachments:
0001-Add-some-backward-compatible-support-for-reading-old.patch,
0002-Make-sure-that-a-connection-always-uses-a-protocol.patch,
0003-Change-the-code-used-when-serializing-HTableDescript.patch, 5204-92.txt,
5204-trunk.txt, 5204.addendum
Attached are 3 patches that are necessary to allow compatibility between
HBase 0.90.x (and previous releases) and HBase 0.92.0.
First of all, I'm well aware that 0.92.0 RC4 has been thumbed up by a lot of
people and would probably wind up being released as 0.92.0 tomorrow, so I
sincerely apologize for creating this issue so late in the process. I spent
a lot of time trying to work around the quirks of 0.92 but once I realized
that with a few very quasi-trivial changes compatibility would be made
significantly easier, I immediately sent these 3 patches to Stack, who
suggested I create this issue.
The first patch is required as without it clients sending a 0.90-style RPC to
a 0.92-style server causes the server to die uncleanly. It seems that 0.92
ships with {{\-XX:OnOutOfMemoryError=kill \-9 %p}}, and when a 0.92 server
fails to deserialize a 0.90-style RPC, it attempts to allocate a large buffer
because it doesn't read fields of 0.90-style RPCs properly. This allocation
attempt immediately triggers an OOME, which causes the JVM to die abruptly of
a {{SIGKILL}}. So whenever a 0.90.x client attempts to connect to HBase, it
kills whichever RS is hosting the {{\-ROOT-}} region.
The second patch fixes a bug introduced by HBASE-2002, which added support
for letting clients specify what protocol they want to speak. If a client
doesn't properly specify what protocol to use, the connection's {{protocol}}
field will be left {{null}}, which causes any subsequent RPC on that
connection to trigger an NPE in the server, even though the connection was
successfully established from the client's point of view. The fix is to
simply give the connection a default protocol, by assuming the client meant
to speak to a RegionServer.
The third patch fixes an oversight that slipped in HBASE-451, where a change
to {{HbaseObjectWritable}} caused all the codes used to serialize
{{Writables}} to shift by one. This was carefully avoided in other changes
such as HBASE-1502, which cleanly removed entries for {{HMsg}} and
{{HMsg[]}}, so I don't think this breakage in HBASE-451 was intended.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5223) TestMetaReaderEditor is missing call to CatalogTracker.stop()
[ https://issues.apache.org/jira/browse/HBASE-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188779#comment-13188779 ] Zhihong Yu commented on HBASE-5223: --- Integrated to TRUNK. TestMetaReaderEditor is missing call to CatalogTracker.stop() - Key: HBASE-5223 URL: https://issues.apache.org/jira/browse/HBASE-5223 Project: HBase Issue Type: Test Reporter: Zhihong Yu Fix For: 0.94.0, 0.92.1 Attachments: 5223.txt I noticed that TestMetaReaderEditor hung on 0.92 Jenkins builds - see https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/249/console It turns out that CatalogTracker.stop() is missing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files before compaction selection
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188780#comment-13188780 ] Phabricator commented on HBASE-5199: khemani has commented on the revision [jira][HBASE-5199] Delete out of TTL store files before compaction selection . INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1040 should Readers be notified via notifyChangeReadersObservers()? src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1016 Can this be done inside a readLock() (for most of the compactions no storefiles will actually get removed) REVISION DETAIL https://reviews.facebook.net/D1311 Delete out of TTL store files before compaction selection - Key: HBASE-5199 URL: https://issues.apache.org/jira/browse/HBASE-5199 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions. In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5204) Backward compatibility fixes for 0.92
[
https://issues.apache.org/jira/browse/HBASE-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188778#comment-13188778
]
stack commented on HBASE-5204:
--
And, this issue can't go into 0.92 because it will break rolling restart (for
those accessing codes that have been reordered). Can go into trunk.
Backward compatibility fixes for 0.92
-
Key: HBASE-5204
URL: https://issues.apache.org/jira/browse/HBASE-5204
Project: HBase
Issue Type: Bug
Components: ipc
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Blocker
Labels: backwards-compatibility
Fix For: 0.92.0
Attachments:
0001-Add-some-backward-compatible-support-for-reading-old.patch,
0002-Make-sure-that-a-connection-always-uses-a-protocol.patch,
0003-Change-the-code-used-when-serializing-HTableDescript.patch, 5204-92.txt,
5204-trunk.txt, 5204.addendum
Attached are 3 patches that are necessary to allow compatibility between
HBase 0.90.x (and previous releases) and HBase 0.92.0.
First of all, I'm well aware that 0.92.0 RC4 has been thumbed up by a lot of
people and would probably wind up being released as 0.92.0 tomorrow, so I
sincerely apologize for creating this issue so late in the process. I spent
a lot of time trying to work around the quirks of 0.92 but once I realized
that with a few very quasi-trivial changes compatibility would be made
significantly easier, I immediately sent these 3 patches to Stack, who
suggested I create this issue.
The first patch is required as without it clients sending a 0.90-style RPC to
a 0.92-style server causes the server to die uncleanly. It seems that 0.92
ships with {{\-XX:OnOutOfMemoryError=kill \-9 %p}}, and when a 0.92 server
fails to deserialize a 0.90-style RPC, it attempts to allocate a large buffer
because it doesn't read fields of 0.90-style RPCs properly. This allocation
attempt immediately triggers an OOME, which causes the JVM to die abruptly of
a {{SIGKILL}}. So whenever a 0.90.x client attempts to connect to HBase, it
kills whichever RS is hosting the {{\-ROOT-}} region.
The second patch fixes a bug introduced by HBASE-2002, which added support
for letting clients specify what protocol they want to speak. If a client
doesn't properly specify what protocol to use, the connection's {{protocol}}
field will be left {{null}}, which causes any subsequent RPC on that
connection to trigger an NPE in the server, even though the connection was
successfully established from the client's point of view. The fix is to
simply give the connection a default protocol, by assuming the client meant
to speak to a RegionServer.
The third patch fixes an oversight that slipped in HBASE-451, where a change
to {{HbaseObjectWritable}} caused all the codes used to serialize
{{Writables}} to shift by one. This was carefully avoided in other changes
such as HBASE-1502, which cleanly removed entries for {{HMsg}} and
{{HMsg[]}}, so I don't think this breakage in HBASE-451 was intended.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
[
https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188784#comment-13188784
]
Mikhail Bautin commented on HBASE-3976:
---
The last commit I can see for this JIRA is the following:
{quote}
Author: stack stack@13f79535-47bb-0310-9956-ffa450edef68
Date: Tue Jun 14 16:47:37 2011
HBASE-3976 Disable Block Cache On Compactions -- REVERT
{quote}
This makes me think this path was actually reverted in trunk. Besides, when
Jonathan's HBASE-4422 (CacheConfig) went in, it completely changed the way this
feature would be implemented. This is why it is not possible to verify whether
the patch has been applied or not anymore from looking at the current version
of Store.java in trunk. I can't say off the top of my head whether we cache
blocks on write during compactions in HBase trunk these days with the new logic
implemented by HBASE-4422 in effect. However, I am planning to write a new unit
test for both trunk and 89-fb to verify that, and address this JIRA if
necessary in a consistent way across the two branches.
Disable Block Cache On Compactions
--
Key: HBASE-3976
URL: https://issues.apache.org/jira/browse/HBASE-3976
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 0.90.3
Reporter: Karthick Sankarachary
Assignee: Nicolas Spiegelberg
Priority: Minor
Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch,
HBASE-3976.patch
Is there a good reason to believe that caching blocks during compactions is
beneficial? Currently, if block cache is enabled on a certain family, then
every time it's compacted, we load all of its blocks into the (LRU) cache, at
the expense of the legitimately hot ones.
As a matter of fact, this concern was raised earlier in HBASE-1597, which
rightly points out that, we should not bog down the LRU with unneccessary
blocks during compaction. Even though that issue has been marked as fixed,
it looks like it ought to be reopened.
Should we err on the side of caution and not cache blocks during compactions
period (as illustrated in the attached patch)? Or, can we be selectively
aggressive about what blocks do get cached during compaction (e.g., only
cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3421) Very wide rows -- 30M plus -- cause us OOME
[ https://issues.apache.org/jira/browse/HBASE-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HBASE-3421: --- Release Note: A new config parameter, hbase.hstore.compaction.kv.max, has been added to limit the number of rows processed in each iteration of the internal compaction code. Default value is 10. was: A new config parameter, hbase.hstore.compaction.kv.max, has been added to limit the number of rows scanner returns in next(). Default value is 10. Very wide rows -- 30M plus -- cause us OOME --- Key: HBASE-3421 URL: https://issues.apache.org/jira/browse/HBASE-3421 Project: HBase Issue Type: Bug Affects Versions: 0.90.0 Reporter: stack Assignee: Nate Putnam Fix For: 0.90.5 Attachments: 3421.addendum, HBASE-3421.patch, HBASE-34211-v2.patch, HBASE-34211-v3.patch, HBASE-34211-v4.patch From the list, see 'jvm oom' in http://mail-archives.apache.org/mod_mbox/hbase-user/201101.mbox/browser, it looks like wide rows -- 30M or so -- causes OOME during compaction. We should check it out. Can the scanner used during compactions use the 'limit' when nexting? If so, this should save our OOME'ing (or, we need to add to the next a max size rather than count of KVs). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2600) Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid
[ https://issues.apache.org/jira/browse/HBASE-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188795#comment-13188795 ] Lars Hofhansl commented on HBASE-2600: -- @Alex: What the problem with TestMetaMigrationRemovingHTD? The tar file with that version of HBase is in ./src/test/data/hbase-4388-root.dir.tgz. I am actually not sure what it means if that test fails. @Stack: That seems to be the last stumbling block to get a successful test run. I assume this test failing indicates something bad (that we cannot migrate meta with this patch applied). Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid Key: HBASE-2600 URL: https://issues.apache.org/jira/browse/HBASE-2600 Project: HBase Issue Type: Bug Reporter: stack Assignee: Alex Newman Attachments: 0001-Changed-regioninfo-format-to-use-endKey-instead-of-s.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v2.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v4.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v6.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v7.2.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8.1, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v9.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen.patch, 2600-trunk-01-17.txt, jenkins.pdf This is an idea that Ryan and I have been kicking around on and off for a while now. If regionnames were made of tablename+endrow instead of tablename+startrow, then in the metatables, doing a search for the region that contains the wanted row, we'd just have to open a scanner using passed row and the first row found by the scan would be that of the region we need (If offlined parent, we'd have to scan to the next row). If we redid the meta tables in this format, we'd be using an access that is natural to hbase, a scan as opposed to the perverse, expensive getClosestRowBefore we currently have that has to walk backward in meta finding a containing region. This issue is about changing the way we name regions. If we were using scans, prewarming client cache would be near costless (as opposed to what we'll currently have to do which is first a getClosestRowBefore and then a scan from the closestrowbefore forward). Converting to the new method, we'd have to run a migration on startup changing the content in meta. Up to this, the randomid component of a region name has been the timestamp of region creation. HBASE-2531 32-bit encoding of regionnames waaay too susceptible to hash clashes proposes changing the randomid so that it contains actual name of the directory in the filesystem that hosts the region. If we had this in place, I think it would help with the migration to this new way of doing the meta because as is, the region name in fs is a hash of regionname... changing the format of the regionname would mean we generate a different hash... so we'd need hbase-2531 to be in place before we could do this change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-2600) Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid
[ https://issues.apache.org/jira/browse/HBASE-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188795#comment-13188795 ] Lars Hofhansl edited comment on HBASE-2600 at 1/18/12 11:17 PM: @Alex: What is the problem with TestMetaMigrationRemovingHTD? The tar file with that version of HBase is in ./src/test/data/hbase-4388-root.dir.tgz. I am actually not sure what it means if that test fails. @Stack: That seems to be the last stumbling block to get a successful test run. I assume this test failing indicates something bad (that we cannot migrate meta with this patch applied). was (Author: lhofhansl): @Alex: What the problem with TestMetaMigrationRemovingHTD? The tar file with that version of HBase is in ./src/test/data/hbase-4388-root.dir.tgz. I am actually not sure what it means if that test fails. @Stack: That seems to be the last stumbling block to get a successful test run. I assume this test failing indicates something bad (that we cannot migrate meta with this patch applied). Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid Key: HBASE-2600 URL: https://issues.apache.org/jira/browse/HBASE-2600 Project: HBase Issue Type: Bug Reporter: stack Assignee: Alex Newman Attachments: 0001-Changed-regioninfo-format-to-use-endKey-instead-of-s.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v2.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v4.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v6.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v7.2.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8.1, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v9.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen.patch, 2600-trunk-01-17.txt, jenkins.pdf This is an idea that Ryan and I have been kicking around on and off for a while now. If regionnames were made of tablename+endrow instead of tablename+startrow, then in the metatables, doing a search for the region that contains the wanted row, we'd just have to open a scanner using passed row and the first row found by the scan would be that of the region we need (If offlined parent, we'd have to scan to the next row). If we redid the meta tables in this format, we'd be using an access that is natural to hbase, a scan as opposed to the perverse, expensive getClosestRowBefore we currently have that has to walk backward in meta finding a containing region. This issue is about changing the way we name regions. If we were using scans, prewarming client cache would be near costless (as opposed to what we'll currently have to do which is first a getClosestRowBefore and then a scan from the closestrowbefore forward). Converting to the new method, we'd have to run a migration on startup changing the content in meta. Up to this, the randomid component of a region name has been the timestamp of region creation. HBASE-2531 32-bit encoding of regionnames waaay too susceptible to hash clashes proposes changing the randomid so that it contains actual name of the directory in the filesystem that hosts the region. If we had this in place, I think it would help with the migration to this new way of doing the meta because as is, the region name in fs is a hash of regionname... changing the format of the regionname would mean we generate a different hash... so we'd need hbase-2531 to be in place before we could do this change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2600) Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid
[ https://issues.apache.org/jira/browse/HBASE-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188798#comment-13188798 ] Zhihong Yu commented on HBASE-2600: --- I think we should shift our attention from TestMetaMigrationRemovingHTD to creating a new migration test. The reason is that 0.90 to 0.94 is not a supported migration path. 1. A tar ball of 0.92 HBase should be generated. 2. Verify that we can migrate from 0.92 .META. table to the new format. Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid Key: HBASE-2600 URL: https://issues.apache.org/jira/browse/HBASE-2600 Project: HBase Issue Type: Bug Reporter: stack Assignee: Alex Newman Attachments: 0001-Changed-regioninfo-format-to-use-endKey-instead-of-s.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v2.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v4.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v6.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v7.2.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8.1, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v9.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen.patch, 2600-trunk-01-17.txt, jenkins.pdf This is an idea that Ryan and I have been kicking around on and off for a while now. If regionnames were made of tablename+endrow instead of tablename+startrow, then in the metatables, doing a search for the region that contains the wanted row, we'd just have to open a scanner using passed row and the first row found by the scan would be that of the region we need (If offlined parent, we'd have to scan to the next row). If we redid the meta tables in this format, we'd be using an access that is natural to hbase, a scan as opposed to the perverse, expensive getClosestRowBefore we currently have that has to walk backward in meta finding a containing region. This issue is about changing the way we name regions. If we were using scans, prewarming client cache would be near costless (as opposed to what we'll currently have to do which is first a getClosestRowBefore and then a scan from the closestrowbefore forward). Converting to the new method, we'd have to run a migration on startup changing the content in meta. Up to this, the randomid component of a region name has been the timestamp of region creation. HBASE-2531 32-bit encoding of regionnames waaay too susceptible to hash clashes proposes changing the randomid so that it contains actual name of the directory in the filesystem that hosts the region. If we had this in place, I think it would help with the migration to this new way of doing the meta because as is, the region name in fs is a hash of regionname... changing the format of the regionname would mean we generate a different hash... so we'd need hbase-2531 to be in place before we could do this change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2600) Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid
[ https://issues.apache.org/jira/browse/HBASE-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188805#comment-13188805 ] Lars Hofhansl commented on HBASE-2600: -- Oh? Did now know that. If that's the case, I agree we should remove that test and create a similar test for this issue. Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid Key: HBASE-2600 URL: https://issues.apache.org/jira/browse/HBASE-2600 Project: HBase Issue Type: Bug Reporter: stack Assignee: Alex Newman Attachments: 0001-Changed-regioninfo-format-to-use-endKey-instead-of-s.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v2.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v4.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v6.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v7.2.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8.1, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v9.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen.patch, 2600-trunk-01-17.txt, jenkins.pdf This is an idea that Ryan and I have been kicking around on and off for a while now. If regionnames were made of tablename+endrow instead of tablename+startrow, then in the metatables, doing a search for the region that contains the wanted row, we'd just have to open a scanner using passed row and the first row found by the scan would be that of the region we need (If offlined parent, we'd have to scan to the next row). If we redid the meta tables in this format, we'd be using an access that is natural to hbase, a scan as opposed to the perverse, expensive getClosestRowBefore we currently have that has to walk backward in meta finding a containing region. This issue is about changing the way we name regions. If we were using scans, prewarming client cache would be near costless (as opposed to what we'll currently have to do which is first a getClosestRowBefore and then a scan from the closestrowbefore forward). Converting to the new method, we'd have to run a migration on startup changing the content in meta. Up to this, the randomid component of a region name has been the timestamp of region creation. HBASE-2531 32-bit encoding of regionnames waaay too susceptible to hash clashes proposes changing the randomid so that it contains actual name of the directory in the filesystem that hosts the region. If we had this in place, I think it would help with the migration to this new way of doing the meta because as is, the region name in fs is a hash of regionname... changing the format of the regionname would mean we generate a different hash... so we'd need hbase-2531 to be in place before we could do this change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5204) Backward compatibility fixes for 0.92
[
https://issues.apache.org/jira/browse/HBASE-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188804#comment-13188804
]
Zhihong Yu commented on HBASE-5204:
---
Applying the addendum to TRUNK only would make codes for classes out of sync
among 0.90, 0.92 and 0.94
Confusion would be higher.
Backward compatibility fixes for 0.92
-
Key: HBASE-5204
URL: https://issues.apache.org/jira/browse/HBASE-5204
Project: HBase
Issue Type: Bug
Components: ipc
Reporter: Benoit Sigoure
Assignee: Benoit Sigoure
Priority: Blocker
Labels: backwards-compatibility
Fix For: 0.92.0
Attachments:
0001-Add-some-backward-compatible-support-for-reading-old.patch,
0002-Make-sure-that-a-connection-always-uses-a-protocol.patch,
0003-Change-the-code-used-when-serializing-HTableDescript.patch, 5204-92.txt,
5204-trunk.txt, 5204.addendum
Attached are 3 patches that are necessary to allow compatibility between
HBase 0.90.x (and previous releases) and HBase 0.92.0.
First of all, I'm well aware that 0.92.0 RC4 has been thumbed up by a lot of
people and would probably wind up being released as 0.92.0 tomorrow, so I
sincerely apologize for creating this issue so late in the process. I spent
a lot of time trying to work around the quirks of 0.92 but once I realized
that with a few very quasi-trivial changes compatibility would be made
significantly easier, I immediately sent these 3 patches to Stack, who
suggested I create this issue.
The first patch is required as without it clients sending a 0.90-style RPC to
a 0.92-style server causes the server to die uncleanly. It seems that 0.92
ships with {{\-XX:OnOutOfMemoryError=kill \-9 %p}}, and when a 0.92 server
fails to deserialize a 0.90-style RPC, it attempts to allocate a large buffer
because it doesn't read fields of 0.90-style RPCs properly. This allocation
attempt immediately triggers an OOME, which causes the JVM to die abruptly of
a {{SIGKILL}}. So whenever a 0.90.x client attempts to connect to HBase, it
kills whichever RS is hosting the {{\-ROOT-}} region.
The second patch fixes a bug introduced by HBASE-2002, which added support
for letting clients specify what protocol they want to speak. If a client
doesn't properly specify what protocol to use, the connection's {{protocol}}
field will be left {{null}}, which causes any subsequent RPC on that
connection to trigger an NPE in the server, even though the connection was
successfully established from the client's point of view. The fix is to
simply give the connection a default protocol, by assuming the client meant
to speak to a RegionServer.
The third patch fixes an oversight that slipped in HBASE-451, where a change
to {{HbaseObjectWritable}} caused all the codes used to serialize
{{Writables}} to shift by one. This was carefully avoided in other changes
such as HBASE-1502, which cleanly removed entries for {{HMsg}} and
{{HMsg[]}}, so I don't think this breakage in HBASE-451 was intended.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5150) Failure in a thread may not fail a test, clean up log splitting test
[ https://issues.apache.org/jira/browse/HBASE-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-5150: --- Resolution: Fixed Status: Resolved (was: Patch Available) Failure in a thread may not fail a test, clean up log splitting test Key: HBASE-5150 URL: https://issues.apache.org/jira/browse/HBASE-5150 Project: HBase Issue Type: Test Affects Versions: 0.94.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.94.0 Attachments: hbase-5150.txt, hbase_5150_v3.patch This is to clean up some tests for HBASE-5081. The Assert.fail method in a separate thread will terminate the thread, but may not fail the test. We can use callable, so that we can get the error in getting the result. Some documentation to explain the test will be helpful too. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-2600) Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid
[ https://issues.apache.org/jira/browse/HBASE-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188805#comment-13188805 ] Lars Hofhansl edited comment on HBASE-2600 at 1/18/12 11:37 PM: Oh? Did not know that. If that's the case, I agree we should remove that test and create a similar test for this issue. was (Author: lhofhansl): Oh? Did now know that. If that's the case, I agree we should remove that test and create a similar test for this issue. Change how we do meta tables; from tablename+STARTROW+randomid to instead, tablename+ENDROW+randomid Key: HBASE-2600 URL: https://issues.apache.org/jira/browse/HBASE-2600 Project: HBase Issue Type: Bug Reporter: stack Assignee: Alex Newman Attachments: 0001-Changed-regioninfo-format-to-use-endKey-instead-of-s.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v2.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v4.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v6.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v7.2.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v8.1, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen-v9.patch, 0001-HBASE-2600.-Change-how-we-do-meta-tables-from-tablen.patch, 2600-trunk-01-17.txt, jenkins.pdf This is an idea that Ryan and I have been kicking around on and off for a while now. If regionnames were made of tablename+endrow instead of tablename+startrow, then in the metatables, doing a search for the region that contains the wanted row, we'd just have to open a scanner using passed row and the first row found by the scan would be that of the region we need (If offlined parent, we'd have to scan to the next row). If we redid the meta tables in this format, we'd be using an access that is natural to hbase, a scan as opposed to the perverse, expensive getClosestRowBefore we currently have that has to walk backward in meta finding a containing region. This issue is about changing the way we name regions. If we were using scans, prewarming client cache would be near costless (as opposed to what we'll currently have to do which is first a getClosestRowBefore and then a scan from the closestrowbefore forward). Converting to the new method, we'd have to run a migration on startup changing the content in meta. Up to this, the randomid component of a region name has been the timestamp of region creation. HBASE-2531 32-bit encoding of regionnames waaay too susceptible to hash clashes proposes changing the randomid so that it contains actual name of the directory in the filesystem that hosts the region. If we had this in place, I think it would help with the migration to this new way of doing the meta because as is, the region name in fs is a hash of regionname... changing the format of the regionname would mean we generate a different hash... so we'd need hbase-2531 to be in place before we could do this change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5010) Filter HFiles based on TTL
[
https://issues.apache.org/jira/browse/HBASE-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188812#comment-13188812
]
Mikhail Bautin commented on HBASE-5010:
---
@Stack: there a small item pending here. The 89-fb patch contains more testing
for the compaction case, which has to be ported to trunk.
@Lars: I will work with Prakash to ensure this does not break HBASE-4721. I
believe HBASE-4721 is only used for a replication approach that is still in
development.
I think we'll keep this JIRA open until the two above items are resolved,
unless anyone objects.
Filter HFiles based on TTL
--
Key: HBASE-5010
URL: https://issues.apache.org/jira/browse/HBASE-5010
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Zhihong Yu
Attachments: 5010.patch, D1017.1.patch, D1017.2.patch, D909.1.patch,
D909.2.patch, D909.3.patch, D909.4.patch, D909.5.patch, D909.6.patch
In ScanWildcardColumnTracker we have
{code:java}
this.oldestStamp = EnvironmentEdgeManager.currentTimeMillis() - ttl;
...
private boolean isExpired(long timestamp) {
return timestamp oldestStamp;
}
{code}
but this time range filtering does not participate in HFile selection. In one
real case this caused next() calls to time out because all KVs in a table got
expired, but next() had to iterate over the whole table to find that out. We
should be able to filter out those HFiles right away. I think a reasonable
approach is to add a default timerange filter to every scan for a CF with a
finite TTL and utilize existing filtering in
StoreFile.Reader.passesTimerangeFilter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2315) BookKeeper for write-ahead logging
[
https://issues.apache.org/jira/browse/HBASE-2315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188821#comment-13188821
]
Zhihong Yu commented on HBASE-2315:
---
@Flavio:
I searched Omid code and didn't find BookKeeperLogWriter being used.
If you still prefer to add BookKeeperLog{Writer,Reader} to HBase, please
provide an updated patch (including pom.xml).
BookKeeper for write-ahead logging
--
Key: HBASE-2315
URL: https://issues.apache.org/jira/browse/HBASE-2315
Project: HBase
Issue Type: New Feature
Components: regionserver
Reporter: Flavio Junqueira
Attachments: HBASE-2315.patch, bookkeeperOverview.pdf,
zookeeper-dev-bookkeeper.jar
BookKeeper, a contrib of the ZooKeeper project, is a fault tolerant and high
throughput write-ahead logging service. This issue provides an implementation
of write-ahead logging for hbase using BookKeeper. Apart from expected
throughput improvements, BookKeeper also has stronger durability guarantees
compared to the implementation currently used by hbase.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2315) BookKeeper for write-ahead logging
[ https://issues.apache.org/jira/browse/HBASE-2315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188822#comment-13188822 ] Amr Awadallah commented on HBASE-2315: -- I am out of office in an all-day business meeting today and tomorrow, I will not be checking email. If this is urgent then please call my cell phone (or send an SMS), otherwise I will reply to your email when I get back. Thanks for your patience, -- amr BookKeeper for write-ahead logging -- Key: HBASE-2315 URL: https://issues.apache.org/jira/browse/HBASE-2315 Project: HBase Issue Type: New Feature Components: regionserver Reporter: Flavio Junqueira Attachments: HBASE-2315.patch, bookkeeperOverview.pdf, zookeeper-dev-bookkeeper.jar BookKeeper, a contrib of the ZooKeeper project, is a fault tolerant and high throughput write-ahead logging service. This issue provides an implementation of write-ahead logging for hbase using BookKeeper. Apart from expected throughput improvements, BookKeeper also has stronger durability guarantees compared to the implementation currently used by hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files before compaction selection
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188823#comment-13188823 ] Phabricator commented on HBASE-5199: tedyu has commented on the revision [jira][HBASE-5199] Delete out of TTL store files before compaction selection . INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1040 Currently notifyChangedReadersObservers() is called outside lock.writeLock(). Please add the call after line 1044. REVISION DETAIL https://reviews.facebook.net/D1311 Delete out of TTL store files before compaction selection - Key: HBASE-5199 URL: https://issues.apache.org/jira/browse/HBASE-5199 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions. In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4268) Add utility to entirely clear out ZK
[ https://issues.apache.org/jira/browse/HBASE-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188831#comment-13188831 ] Zhihong Yu commented on HBASE-4268: --- @Todd: Are you working on this feature ? I don't see attachment. Thanks Add utility to entirely clear out ZK Key: HBASE-4268 URL: https://issues.apache.org/jira/browse/HBASE-4268 Project: HBase Issue Type: New Feature Components: scripts Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon In disaster scenarios, sometimes some cruft is left over in ZK, when it would be better to do a truely clean startup. We should add a script which allows the admin to clear out ZK while the cluster is down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5228) [REST] Rip out transform feature
[ https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188832#comment-13188832 ] Andrew Purtell commented on HBASE-5228: --- Jack confirmed in private correspondence that this can come out. I'd like to do it ASAP as part of 0.92 (and any further 0.90 release) so we don't pick up a user of this misfeature. [REST] Rip out transform feature -- Key: HBASE-5228 URL: https://issues.apache.org/jira/browse/HBASE-5228 Project: HBase Issue Type: Bug Affects Versions: 0.92.0, 0.94.0, 0.90.5 Reporter: Andrew Purtell Assignee: Andrew Purtell Attachments: HBASE-5228.patch The 'transform' feature, where REST can be instructed, via a table attribute, to apply a transformation (e.g. base64 encoding or decoding) to a (sub)set of column values before serving them up to a client or storing them into HBase, was added some time ago at the request of Jack Levin. I have since come to regret it, it was not a well thought out feature: - This is really an application concern. - It adds significant overhead to request processing: Periodically a HBaseAdmin is used to retrieve the table descriptor, in order to scan through table attributes for transformation directives. I think it is best to rip it out, its a real problem area, and REST should be no more concerned about data formats than the Java API. I doubt anyone uses this, not even Jack. Will need to follow up with him to confirm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files before compaction selection
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188840#comment-13188840 ] Phabricator commented on HBASE-5199: khemani has commented on the revision [jira][HBASE-5199] Delete out of TTL store files before compaction selection . INLINE COMMENTS src/main/java/org/apache/hadoop/hbase/regionserver/Store.java:1016 find out the set of files to be deleted in a read lock. If that set is non-empty, then acquire the write lock and remove the to-be-deleted set of files (if they still exist) from the storefiles set. REVISION DETAIL https://reviews.facebook.net/D1311 Delete out of TTL store files before compaction selection - Key: HBASE-5199 URL: https://issues.apache.org/jira/browse/HBASE-5199 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions. In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5228) [REST] Rip out transform feature
[ https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188842#comment-13188842 ] stack commented on HBASE-5228: -- Can this be in 0.92.1 Andrew? I'd start a 0.92.1 candidate immediately after 0.92.0 release? [REST] Rip out transform feature -- Key: HBASE-5228 URL: https://issues.apache.org/jira/browse/HBASE-5228 Project: HBase Issue Type: Bug Affects Versions: 0.92.0, 0.94.0, 0.90.5 Reporter: Andrew Purtell Assignee: Andrew Purtell Attachments: HBASE-5228.patch The 'transform' feature, where REST can be instructed, via a table attribute, to apply a transformation (e.g. base64 encoding or decoding) to a (sub)set of column values before serving them up to a client or storing them into HBase, was added some time ago at the request of Jack Levin. I have since come to regret it, it was not a well thought out feature: - This is really an application concern. - It adds significant overhead to request processing: Periodically a HBaseAdmin is used to retrieve the table descriptor, in order to scan through table attributes for transformation directives. I think it is best to rip it out, its a real problem area, and REST should be no more concerned about data formats than the Java API. I doubt anyone uses this, not even Jack. Will need to follow up with him to confirm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5228) [REST] Rip out transform feature
[ https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188845#comment-13188845 ] Zhihong Yu commented on HBASE-5228: --- @Andy: Do you want Hadoop QA to run the patch through tests ? [REST] Rip out transform feature -- Key: HBASE-5228 URL: https://issues.apache.org/jira/browse/HBASE-5228 Project: HBase Issue Type: Bug Affects Versions: 0.92.0, 0.94.0, 0.90.5 Reporter: Andrew Purtell Assignee: Andrew Purtell Attachments: HBASE-5228.patch The 'transform' feature, where REST can be instructed, via a table attribute, to apply a transformation (e.g. base64 encoding or decoding) to a (sub)set of column values before serving them up to a client or storing them into HBase, was added some time ago at the request of Jack Levin. I have since come to regret it, it was not a well thought out feature: - This is really an application concern. - It adds significant overhead to request processing: Periodically a HBaseAdmin is used to retrieve the table descriptor, in order to scan through table attributes for transformation directives. I think it is best to rip it out, its a real problem area, and REST should be no more concerned about data formats than the Java API. I doubt anyone uses this, not even Jack. Will need to follow up with him to confirm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5228) [REST] Rip out transform feature
[
https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188848#comment-13188848
]
Zhihong Yu commented on HBASE-5228:
---
{code}
HBaseAdmin admin = new HBaseAdmin(servlet.getConfiguration());
-return admin.tableExists(table);
+boolean result = admin.tableExists(table);
+admin.close();
{code}
Looks like admin.close() should be enclosed in finally block.
[REST] Rip out transform feature
--
Key: HBASE-5228
URL: https://issues.apache.org/jira/browse/HBASE-5228
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0, 0.90.5
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Attachments: HBASE-5228.patch
The 'transform' feature, where REST can be instructed, via a table attribute,
to apply a transformation (e.g. base64 encoding or decoding) to a (sub)set of
column values before serving them up to a client or storing them into HBase,
was added some time ago at the request of Jack Levin. I have since come to
regret it, it was not a well thought out feature:
- This is really an application concern.
- It adds significant overhead to request processing: Periodically a
HBaseAdmin is used to retrieve the table descriptor, in order to scan through
table attributes for transformation directives.
I think it is best to rip it out, its a real problem area, and REST should be
no more concerned about data formats than the Java API.
I doubt anyone uses this, not even Jack. Will need to follow up with him to
confirm.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5229) Support atomic region operations
Support atomic region operations Key: HBASE-5229 URL: https://issues.apache.org/jira/browse/HBASE-5229 Project: HBase Issue Type: New Feature Components: client, regionserver Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 5229.txt As discussed (at length) on the dev mailing list with the HBASE-3584 and HBASE-5203 committed, supporting atomic cross row transactions within a region becomes simple. I am aware of the hesitation about the usefulness of this feature, but we have to start somewhere. Let's use this jira for discussion, I'll attach a patch (with tests) momentarily to make this concrete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4268) Add utility to entirely clear out ZK
[ https://issues.apache.org/jira/browse/HBASE-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reassigned HBASE-4268: -- Assignee: (was: Todd Lipcon) Nope, I got distracted away from this task a few months ago. I can't seem to find the code that I'd started, either. Add utility to entirely clear out ZK Key: HBASE-4268 URL: https://issues.apache.org/jira/browse/HBASE-4268 Project: HBase Issue Type: New Feature Components: scripts Affects Versions: 0.92.0 Reporter: Todd Lipcon In disaster scenarios, sometimes some cruft is left over in ZK, when it would be better to do a truely clean startup. We should add a script which allows the admin to clear out ZK while the cluster is down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5229) Support atomic region operations
[ https://issues.apache.org/jira/browse/HBASE-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5229: - Attachment: 5229.txt Initial patch. It works as advertised. I couldn't think of good names for the classes (RegionMutation is not a good name). Support atomic region operations Key: HBASE-5229 URL: https://issues.apache.org/jira/browse/HBASE-5229 Project: HBase Issue Type: New Feature Components: client, regionserver Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 5229.txt As discussed (at length) on the dev mailing list with the HBASE-3584 and HBASE-5203 committed, supporting atomic cross row transactions within a region becomes simple. I am aware of the hesitation about the usefulness of this feature, but we have to start somewhere. Let's use this jira for discussion, I'll attach a patch (with tests) momentarily to make this concrete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5228) [REST] Rip out transform feature
[ https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-5228: -- Fix Version/s: 0.90.6 0.92.1 0.94.0 @Stack: Ok @Ted: Thanks for the patch review, will fix on commit if you think this is good to go in otherwise. [REST] Rip out transform feature -- Key: HBASE-5228 URL: https://issues.apache.org/jira/browse/HBASE-5228 Project: HBase Issue Type: Bug Affects Versions: 0.92.0, 0.94.0, 0.90.5 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.94.0, 0.92.1, 0.90.6 Attachments: HBASE-5228.patch The 'transform' feature, where REST can be instructed, via a table attribute, to apply a transformation (e.g. base64 encoding or decoding) to a (sub)set of column values before serving them up to a client or storing them into HBase, was added some time ago at the request of Jack Levin. I have since come to regret it, it was not a well thought out feature: - This is really an application concern. - It adds significant overhead to request processing: Periodically a HBaseAdmin is used to retrieve the table descriptor, in order to scan through table attributes for transformation directives. I think it is best to rip it out, its a real problem area, and REST should be no more concerned about data formats than the Java API. I doubt anyone uses this, not even Jack. Will need to follow up with him to confirm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5228) [REST] Rip out transform feature
[ https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188856#comment-13188856 ] Zhihong Yu commented on HBASE-5228: --- +1 if tests pass. The current patch wouldn't apply cleanly on TRUNK, FYI [REST] Rip out transform feature -- Key: HBASE-5228 URL: https://issues.apache.org/jira/browse/HBASE-5228 Project: HBase Issue Type: Bug Affects Versions: 0.92.0, 0.94.0, 0.90.5 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.94.0, 0.92.1, 0.90.6 Attachments: HBASE-5228.patch The 'transform' feature, where REST can be instructed, via a table attribute, to apply a transformation (e.g. base64 encoding or decoding) to a (sub)set of column values before serving them up to a client or storing them into HBase, was added some time ago at the request of Jack Levin. I have since come to regret it, it was not a well thought out feature: - This is really an application concern. - It adds significant overhead to request processing: Periodically a HBaseAdmin is used to retrieve the table descriptor, in order to scan through table attributes for transformation directives. I think it is best to rip it out, its a real problem area, and REST should be no more concerned about data formats than the Java API. I doubt anyone uses this, not even Jack. Will need to follow up with him to confirm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4268) Add utility to entirely clear out ZK
[ https://issues.apache.org/jira/browse/HBASE-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188859#comment-13188859 ] Karthik K commented on HBASE-4268: -- I had put in ZOOKEEPER-729 sometime ago, for the exact reason in the past. With zk 3.4.2 , we can delete recursively as : ./bin/zkCli.sh rmr /hbase From the latest comment, seems like the command has been renamed in future releases though: Add utility to entirely clear out ZK Key: HBASE-4268 URL: https://issues.apache.org/jira/browse/HBASE-4268 Project: HBase Issue Type: New Feature Components: scripts Affects Versions: 0.92.0 Reporter: Todd Lipcon In disaster scenarios, sometimes some cruft is left over in ZK, when it would be better to do a truely clean startup. We should add a script which allows the admin to clear out ZK while the cluster is down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5228) [REST] Rip out transform feature
[ https://issues.apache.org/jira/browse/HBASE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188860#comment-13188860 ] Lars Hofhansl commented on HBASE-5228: -- +1 on idea and patch. [REST] Rip out transform feature -- Key: HBASE-5228 URL: https://issues.apache.org/jira/browse/HBASE-5228 Project: HBase Issue Type: Bug Affects Versions: 0.92.0, 0.94.0, 0.90.5 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.94.0, 0.92.1, 0.90.6 Attachments: HBASE-5228.patch The 'transform' feature, where REST can be instructed, via a table attribute, to apply a transformation (e.g. base64 encoding or decoding) to a (sub)set of column values before serving them up to a client or storing them into HBase, was added some time ago at the request of Jack Levin. I have since come to regret it, it was not a well thought out feature: - This is really an application concern. - It adds significant overhead to request processing: Periodically a HBaseAdmin is used to retrieve the table descriptor, in order to scan through table attributes for transformation directives. I think it is best to rip it out, its a real problem area, and REST should be no more concerned about data formats than the Java API. I doubt anyone uses this, not even Jack. Will need to follow up with him to confirm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
[ https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188861#comment-13188861 ] Zhihong Yu commented on HBASE-5179: --- Tests for 0.90 passed for 5179-90v11.patch Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss Key: HBASE-5179 URL: https://issues.apache.org/jira/browse/HBASE-5179 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.2 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.92.0, 0.94.0, 0.90.6 Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch, 5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt, 5179-v4.txt, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v5.patch, hbase-5179v6.patch, hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch If master's processing its failover and ServerShutdownHandler's processing happen concurrently, it may appear following case. 1.master completed splitLogAfterStartup() 2.RegionserverA restarts, and ServerShutdownHandler is processing. 3.master starts to rebuildUserRegions, and RegionserverA is considered as dead server. 4.master starts to assign regions of RegionserverA because it is a dead server by step3. However, when doing step4(assigning region), ServerShutdownHandler may be doing split log, Therefore, it may cause data loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5230) Unit test to ensure compactions don't cache data on write
Unit test to ensure compactions don't cache data on write - Key: HBASE-5230 URL: https://issues.apache.org/jira/browse/HBASE-5230 Project: HBase Issue Type: Test Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Create a unit test for HBASE-3976 (making sure we don't cache data blocks on write during compactions even if cache-on-write is enabled generally enabled). This is because we have very different implementations of HBASE-3976 without HBASE-4422 CacheConfig (on top of 89-fb, created by Liyin) and with CacheConfig (presumably it's there but not sure if it even works, since the patch in HBASE-3976 may not have been committed). We need to create a unit test to verify that we don't cache data blocks on write during compactions, and resolve HBASE-3976 so that this new unit test does not fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files before compaction selection
[ https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188868#comment-13188868 ] Phabricator commented on HBASE-5199: Liyin has commented on the revision [jira][HBASE-5199] Delete out of TTL store files before compaction selection . @khemani: I see your point, which inspire me to a new idea: When holding a read lock for selection compaction candidates, it should also select the expired deletion store file candidates. After when processing completeCompaction logic within the write lock, it will delete the expired store files. So this approach should NOT introduce any potential dead lock risk since there is no read or write lock introduced by this feature at all. @tedyu: Thanks for pointing it out. And no additional notifyChangedReadersObservers() is needed if implementing in this way. REVISION DETAIL https://reviews.facebook.net/D1311 Delete out of TTL store files before compaction selection - Key: HBASE-5199 URL: https://issues.apache.org/jira/browse/HBASE-5199 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Liyin Tang Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch Currently, HBase deletes the out of TTL store files after compaction. We can change the sequence to delete the out of TTL store files before selecting store files for compactions. In this way, HBase can keep deleting the old invalid store files without compaction, and also prevent from unnecessary compactions since the out of TTL store files will be deleted before the compaction selection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5199) Delete out of TTL store files before compaction selection
[
https://issues.apache.org/jira/browse/HBASE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13188891#comment-13188891
]
Zhihong Yu commented on HBASE-5199:
---
This becomes interesting, Liyin.
Currently we have:
{code}
override = region.getCoprocessorHost().preCompactSelection(
this, candidates);
{code}
If I understand your idea correctly, a sub-list of candidates would be
extracted for deletion.
Do you think that we should expose selection of expired store files through
coprocessor ?
Delete out of TTL store files before compaction selection
-
Key: HBASE-5199
URL: https://issues.apache.org/jira/browse/HBASE-5199
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
Attachments: D1311.1.patch, D1311.2.patch, D1311.3.patch
Currently, HBase deletes the out of TTL store files after compaction. We can
change the sequence to delete the out of TTL store files before selecting
store files for compactions.
In this way, HBase can keep deleting the old invalid store files without
compaction, and also prevent from unnecessary compactions since the out of
TTL store files will be deleted before the compaction selection.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
