[jira] [Resolved] (HBASE-4641) Block cache can be mistakenly instantiated on Master
[ https://issues.apache.org/jira/browse/HBASE-4641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Gray resolved HBASE-4641. -- Resolution: Fixed Hadoop Flags: Reviewed Committed v1 to branch and trunk. Thanks guys. Block cache can be mistakenly instantiated on Master Key: HBASE-4641 URL: https://issues.apache.org/jira/browse/HBASE-4641 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0, 0.94.0 Reporter: Jonathan Gray Assignee: Jonathan Gray Priority: Critical Fix For: 0.92.0 Attachments: 4641-suggestion-v3.txt, 4641-v4.txt, HBASE-4641-v1.patch, HBASE-4641-v2.patch After changes in the block cache instantiation over in HBASE-4422, it looks like the HMaster can now end up with a block cache instantiated. Not a huge deal but prevents the process from shutting down properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4687) regionserver may miss zk-heartbeats to master when replaying edits at region open
[ https://issues.apache.org/jira/browse/HBASE-4687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Gray resolved HBASE-4687. -- Resolution: Fixed Fix Version/s: 0.92.0 Hadoop Flags: Reviewed Committed to 92 branch and trunk. regionserver may miss zk-heartbeats to master when replaying edits at region open - Key: HBASE-4687 URL: https://issues.apache.org/jira/browse/HBASE-4687 Project: HBase Issue Type: Bug Reporter: Prakash Khemani Assignee: Prakash Khemani Fix For: 0.92.0 Attachments: 0001-HBASE-4687-regionserver-may-miss-zk-heartbeats-to-ma.patch replayRecoveredEdits() should do another reporter.progress() before returning. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4460) Support running an embedded ThriftServer within a RegionServer
[ https://issues.apache.org/jira/browse/HBASE-4460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Gray resolved HBASE-4460. -- Resolution: Fixed Release Note: Run a ThriftServer embedded within a RegionServer process by setting hbase.regionserver.export.thrift to true. Hadoop Flags: Reviewed Committed to trunk. Thanks for the reviews LarsH and KarthikR. Opened HBASE-4631 to implement some of your suggestions. Support running an embedded ThriftServer within a RegionServer -- Key: HBASE-4460 URL: https://issues.apache.org/jira/browse/HBASE-4460 Project: HBase Issue Type: New Feature Components: regionserver, thrift Reporter: Jonathan Gray Assignee: Jonathan Gray Attachments: HBASE-4460-v1.patch Rather than a separate process, it can be advantageous in some situations for each RegionServer to embed their own ThriftServer. This allows each embedded ThriftServer to short-circuit any queries that should be executed on the local RS and skip the extra hop. This then enables the building of fat Thrift clients that cache region locations and avoid extra hops all together. This JIRA is just about the embedded ThriftServer. Will open others for the rest. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-3417) CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme
[ https://issues.apache.org/jira/browse/HBASE-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Gray resolved HBASE-3417. -- Resolution: Fixed Release Note: StoreFile naming changes from random ascii longs to [0-9a-f] uuids. The same name is used in the tmp and perm location. This name is what gets used for the block name in the cache. Committed to 92 and trunk. Thanks Stack and Mikhail for the speedy reviews. CacheOnWrite is using the temporary output path for block names, need to use a more consistent block naming scheme -- Key: HBASE-3417 URL: https://issues.apache.org/jira/browse/HBASE-3417 Project: HBase Issue Type: Bug Components: io, regionserver Affects Versions: 0.92.0 Reporter: Jonathan Gray Assignee: Jonathan Gray Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3417-redux-v1.patch, HBASE-3417-v1.patch, HBASE-3417-v2.patch, HBASE-3417-v5.patch Currently the block names used in the block cache are built using the filesystem path. However, for cache on write, the path is a temporary output file. The original COW patch actually made some modifications to block naming stuff to make it more consistent but did not do enough. Should add a separate method somewhere for generating block names using some more easily mocked scheme (rather than just raw path as we generate a random unique file name twice, once for tmp and then again when moved into place). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4582) Store.java cleanup (failing TestHeapSize and has warnings)
[ https://issues.apache.org/jira/browse/HBASE-4582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Gray resolved HBASE-4582. -- Resolution: Fixed Hadoop Flags: Reviewed Committed to 92 branch and trunk. Thanks for review stack. Store.java cleanup (failing TestHeapSize and has warnings) -- Key: HBASE-4582 URL: https://issues.apache.org/jira/browse/HBASE-4582 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.92.0, 0.94.0 Reporter: Jonathan Gray Assignee: Jonathan Gray Priority: Minor Fix For: 0.92.0 Attachments: HBASE-4582-FINAL.patch, HBASE-4582-v1-branch92.patch I broke TestHeapSize on 32-bit when I committed HBASE-4422. Fix it here, remove another boolean variable I forgot to delete, and fix a couple warnings while we are at it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Gray resolved HBASE-4422. -- Resolution: Fixed Release Note: Relocates all cache configuration options and constants into a new CacheConfig class. Modifies lots of constructors from Store/StoreFile down to HFile. Hadoop Flags: Reviewed Committed to trunk and branch. Thanks everyone for all the reviews. Thanks Mikhail for all the back and forth! Move block cache parameters and references into single CacheConf class -- Key: HBASE-4422 URL: https://issues.apache.org/jira/browse/HBASE-4422 Project: HBase Issue Type: Improvement Components: io Reporter: Jonathan Gray Assignee: Jonathan Gray Fix For: 0.92.0 Attachments: CacheConfig92-v8.patch, HBASE-4422-FINAL-branch92.patch, HBASE-4422-FINAL-trunk.patch From StoreFile down to HFile, we currently use a boolean argument for each of the various block cache configuration parameters that exist. The number of parameters is going to continue to increase as we look at compressed cache, delta encoding, and more specific L1/L2 configuration. Every new config currently requires changing many constructors because it introduces a new boolean. We should move everything into a single class so that modifications are much less disruptive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4546) Upgrade to ZooKeeper 3.3.2 or 3.3.3
[ https://issues.apache.org/jira/browse/HBASE-4546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Gray resolved HBASE-4546. -- Resolution: Not A Problem Fix Version/s: (was: 0.92.0) Nevermind, I'm looking at a stale pom. We are already on 3.3.3 in 92 and trunk. Upgrade to ZooKeeper 3.3.2 or 3.3.3 --- Key: HBASE-4546 URL: https://issues.apache.org/jira/browse/HBASE-4546 Project: HBase Issue Type: Improvement Components: zookeeper Reporter: Jonathan Gray Assignee: Jonathan Gray HBase is still depending on 3.3.1. There many critical bug fixes in 3.3.2 and two more critical fixes in 3.3.3. We recently tripped on ZOOKEEPER-822 which was fixed in 3.3.2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4534) A new unit test for lazy seek and StoreScanner in general
[ https://issues.apache.org/jira/browse/HBASE-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Gray resolved HBASE-4534. -- Resolution: Fixed Fix Version/s: 0.94.0 Committed to trunk. Nice work Mikhail! A new unit test for lazy seek and StoreScanner in general - Key: HBASE-4534 URL: https://issues.apache.org/jira/browse/HBASE-4534 Project: HBase Issue Type: Test Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Fix For: 0.94.0 A randomized unit test for Gets/Scans (all-row, single-row, multi-row, all-column, single-column, and multi-column). Also all combinations of Bloom filters and compression (NONE vs GZIP) are tested. The unit test flushes multiple StoreFiles with disjoint timestamp ranges and runs various types of queries against them. Currently we are not testing overlapping timestamp ranges. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4477) Ability for an application to store metadata into the transaction log
[ https://issues.apache.org/jira/browse/HBASE-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Gray resolved HBASE-4477. -- Resolution: Fixed Fix Version/s: 0.92.0 Hadoop Flags: Reviewed Committed to trunk and 92 branch. Thanks for all the reviews and feedback all, nice stuff Dhruba. Dhruba opened HBASE-4514 for the follow-up discussed here. Ability for an application to store metadata into the transaction log - Key: HBASE-4477 URL: https://issues.apache.org/jira/browse/HBASE-4477 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.92.0 Attachments: coprocessorPut1.txt, coprocessorPut2.txt, hlogMetadata1.txt mySQL allows an application to store an arbitrary blob along with each transaction in its transaction logs. This JIRA is to have a similar feature request for HBASE. The use case is as follows: An application on one data center A stores a blob of data along with each transaction. A replication software picks up these blobs from the transaction logs in A and hands it to another instance of the same application running on a remote data center B. The application in B is responsible for applying this to the remote Hbase cluster (and also handle conflict resolution if any). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4131) Make the Replication Service pluggable via a standard interface definition
[ https://issues.apache.org/jira/browse/HBASE-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Gray resolved HBASE-4131. -- Resolution: Fixed Committed to trunk. Good work Dhruba and thanks for review Ted! Make the Replication Service pluggable via a standard interface definition -- Key: HBASE-4131 URL: https://issues.apache.org/jira/browse/HBASE-4131 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: 4131-backedout.txt, replicationInterface1.txt, replicationInterface2.txt, replicationInterface3.txt, replicationInterface4.txt The current HBase code supports a replication service that can be used to sync data from from one hbase cluster to another. It would be nice to make it a pluggable interface so that other cross-data-center replication services can be used in conjuction with HBase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4433) avoid extra next (potentially a seek) if done with column/row
[ https://issues.apache.org/jira/browse/HBASE-4433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Gray resolved HBASE-4433. -- Resolution: Fixed Fix Version/s: 0.94.0 Hadoop Flags: Reviewed Good stuff Kannan! Thanks for review Ted. I also looked and I'm +1. Committed to trunk. avoid extra next (potentially a seek) if done with column/row - Key: HBASE-4433 URL: https://issues.apache.org/jira/browse/HBASE-4433 Project: HBase Issue Type: Improvement Reporter: Kannan Muthukkaruppan Assignee: Kannan Muthukkaruppan Fix For: 0.94.0 [Noticed this in 89, but quite likely true of trunk as well.] When we are done with the requested column(s) the code still does an extra next() call before it realizes that it is actually done. This extra next() call could potentially result in an unnecessary extra block load. This is likely to be especially bad for CFs where the KVs are large blobs where each KV may be occupying a block of its own. So the next() can often load a new unrelated block unnecessarily. -- For the simple case of reading say the top-most column in a row in a single file, where each column (KV) was say a block of its own-- it seems that we are reading 3 blocks, instead of 1 block! I am working on a simple patch and with that the number of seeks is down to 2. [There is still an extra seek left. I think there were two levels of extra/unnecessary next() we were doing without actually confirming that the next was needed. One at the StoreScanner/ScanQueryMatcher level which this diff avoids. I think the other is at hfs.next() (at the storefile scanner level) that's happening whenever a HFile scanner servers out a data-- and perhaps that's the additional seek that we need to avoid. But I want to tackle this optimization first as the two issues seem unrelated.] -- The basic idea of the patch I am working on/testing is as follows. The ExplicitColumnTracker currently returns INCLUDE to the ScanQueryMatcher if the KV needs to be included and then if done, only in the the next call it returns the appropriate SEEK_NEXT_COL or SEEK_NEXT_ROW hint. For the cases when ExplicitColumnTracker knows it is done with a particular column/row, the patch attempts to combine the INCLUDE code and done hint into a single match code-- INCLUDE_AND_SEEK_NEXT_COL and INCLUDE_AND_SEEK_NEXT_ROW. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
