[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment
[ https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539865#comment-13539865 ] shen guanpu commented on HBASE-7263: hi Gregory Chanan Does your test just operate one rowkey? How much will it be slower when you put different rowkey? as you mentioned ,you have to wait for MVCC on other rows. Investigate more fine grained locking for checkAndPut/append/increment -- Key: HBASE-7263 URL: https://issues.apache.org/jira/browse/HBASE-7263 Project: HBase Issue Type: Improvement Components: Transactions/MVCC Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Minor HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut: {quote} 1) Waiting for the MVCC to advance for read/updates: the downside is that you have to wait for updates on other rows. 2) Have an MVCC per-row (table configuration): this avoids the unnecessary contention of 1) 3) Transform the read/updates to write-only with rollup on read.. E.g. an increment would just have the number of values to increment. {quote} HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned, is that you have to wait for updates on other rows, since MVCC is per-row. Another option occurred to me that I think is worth investigating: rely on a row-level read/write lock rather than MVCC. Here is pseudo-code for what exists today for read/updates like checkAndPut {code} (1) Acquire RowLock (1a) BeginMVCC + Finish MVCC (2) Begin MVCC (3) Do work (4) Release RowLock (5) Append to WAL (6) Finish MVCC {code} Write-only operations (e.g. puts) are the same, just without step 1a. Now, consider the following instead: {code} (1) Acquire RowLock (1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC) (1b) Grab RowReadLock (new step!) (2) Begin MVCC (3) Do work (4) Release RowLock (5) Append to WAL (6) Finish MVCC (7) Release RowReadLock (new step!) {code} As before, write-only operations are the same, just without step 1a. The difference here is that writes grab a row-level read lock and hold it until the MVCC is completed. The nice property that this gives you is that read/updates can tell when the MVCC is done on a per-row basis, because they can just try to acquire the write-lock which will block until the MVCC is competed for that row in step 7. There is overhead for acquiring the read lock that I need to measure, but it should be small, since there will never be any blocking on acquiring the row-level read lock. This is because the read lock can only block if someone else holds the write lock, but both the write and read lock are only acquired under the row lock. I ran a quick test of this approach over a region (this directly interacts with HRegion, so no client effects): - 30 threads - 5000 increments per thread - 30 columns per increment - Each increment uniformly distributed over 500,000 rows - 5 trials Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms Today: 13950 ms The locking approach: 10877 ms So it looks like an improvement, at least wrt increment. As mentioned, I need to measure the overhead of acquiring the read lock for puts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7188) Move classes into hbase-client
[ https://issues.apache.org/jira/browse/HBASE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539866#comment-13539866 ] stack commented on HBASE-7188: -- Elliott, now client-side, when Invoker does getProtocolVersion, it is reading the local version of the protocol (HBaseClientRPC.getProtocolVersion). Is that right? When does client ever learn the server version? When I set up the protocol, should I not be going to the server to ask its version of the protocol? At least for the server version of this protocol? Or how do you see this versioning and proxying working on protocol Interfaces now? Previous, after getting proxy, we used to do a getProtocolVersion and throw exception if the client and server versions didn't match. We don't do that anymore it seems. Here is old code from 0.94: {code} 17 /** Construct a client-side proxy object that implements the named protocol, 16* talking to a server at the named address. */ 15 public VersionedProtocol getProxy( 14 Class? extends VersionedProtocol protocol, long clientVersion, 13 InetSocketAddress addr, User ticket, 12 Configuration conf, SocketFactory factory, int rpcTimeout) 11 throws IOException { 10 9 VersionedProtocol proxy = 8 (VersionedProtocol) Proxy.newProxyInstance( 7 protocol.getClassLoader(), new Class[] { protocol }, 6 new Invoker(protocol, addr, ticket, conf, factory, rpcTimeout)); 5 if (proxy instanceof VersionedProtocol) { 4 long serverVersion = ((VersionedProtocol)proxy) 3 .getProtocolVersion(protocol.getName(), clientVersion); 2 if (serverVersion != clientVersion) { 1 throw new HBaseRPC.VersionMismatch(protocol.getName(), clientVersion, 0 serverVersion); 1 } 2 } 3 return proxy; 4 } {code} Move classes into hbase-client -- Key: HBASE-7188 URL: https://issues.apache.org/jira/browse/HBASE-7188 Project: HBase Issue Type: Sub-task Components: Client, IPC/RPC Affects Versions: 0.96.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.96.0 Attachments: HBASE-7188-0.patch, HBASE-7188-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7390) Add extra test cases for assignement on the region server and fix the related issues
[ https://issues.apache.org/jira/browse/HBASE-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539867#comment-13539867 ] nkeywal commented on HBASE-7390: It's on a review board, there it a +1 from Nick there. Is there anyone else reviewing it? If not, I will commit next Monday. Add extra test cases for assignement on the region server and fix the related issues Key: HBASE-7390 URL: https://issues.apache.org/jira/browse/HBASE-7390 Project: HBase Issue Type: Bug Components: Region Assignment, regionserver Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Fix For: 0.96.0 Attachments: 7390.v1.patch, 7390.v2.patch, 7390.v3.patch, 7390.v4.patch, 7390.v6.patch, 7390.v7.patch, 7390.v8.patch, assignment_zk_states.jpg We don't have a lot of tests on the region server itself. Here are some. Some of them are failing, feedback welcome. See as well the attached state diagram for the ZK nodes on assignment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment
[ https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539873#comment-13539873 ] shen guanpu commented on HBASE-7263: HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned, is that you have to wait for updates on other rows, since MVCC is per-row. Do you mean it is not per-row for option 2( Have an MVCC per-row (table configuration): this avoids the unnecessary contention of 1)) Investigate more fine grained locking for checkAndPut/append/increment -- Key: HBASE-7263 URL: https://issues.apache.org/jira/browse/HBASE-7263 Project: HBase Issue Type: Improvement Components: Transactions/MVCC Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Minor HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut: {quote} 1) Waiting for the MVCC to advance for read/updates: the downside is that you have to wait for updates on other rows. 2) Have an MVCC per-row (table configuration): this avoids the unnecessary contention of 1) 3) Transform the read/updates to write-only with rollup on read.. E.g. an increment would just have the number of values to increment. {quote} HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned, is that you have to wait for updates on other rows, since MVCC is per-row. Another option occurred to me that I think is worth investigating: rely on a row-level read/write lock rather than MVCC. Here is pseudo-code for what exists today for read/updates like checkAndPut {code} (1) Acquire RowLock (1a) BeginMVCC + Finish MVCC (2) Begin MVCC (3) Do work (4) Release RowLock (5) Append to WAL (6) Finish MVCC {code} Write-only operations (e.g. puts) are the same, just without step 1a. Now, consider the following instead: {code} (1) Acquire RowLock (1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC) (1b) Grab RowReadLock (new step!) (2) Begin MVCC (3) Do work (4) Release RowLock (5) Append to WAL (6) Finish MVCC (7) Release RowReadLock (new step!) {code} As before, write-only operations are the same, just without step 1a. The difference here is that writes grab a row-level read lock and hold it until the MVCC is completed. The nice property that this gives you is that read/updates can tell when the MVCC is done on a per-row basis, because they can just try to acquire the write-lock which will block until the MVCC is competed for that row in step 7. There is overhead for acquiring the read lock that I need to measure, but it should be small, since there will never be any blocking on acquiring the row-level read lock. This is because the read lock can only block if someone else holds the write lock, but both the write and read lock are only acquired under the row lock. I ran a quick test of this approach over a region (this directly interacts with HRegion, so no client effects): - 30 threads - 5000 increments per thread - 30 columns per increment - Each increment uniformly distributed over 500,000 rows - 5 trials Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms Today: 13950 ms The locking approach: 10877 ms So it looks like an improvement, at least wrt increment. As mentioned, I need to measure the overhead of acquiring the read lock for puts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7441) Make ClusterManager in IntegrationTestingUtility pluggable
Liu Shaohui created HBASE-7441: -- Summary: Make ClusterManager in IntegrationTestingUtility pluggable Key: HBASE-7441 URL: https://issues.apache.org/jira/browse/HBASE-7441 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.3 Reporter: Liu Shaohui Priority: Minor After the patch HBASE-7009, we can use ChaosMonkey to test the Hbase cluster. The ClusterManager use ssh to stop/start the rs or master without passwd. To support other cluster manager tool, we need to make clusterManager in IntegrationTestingUtility pluggable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7441) Make ClusterManager in IntegrationTestingUtility pluggable
[ https://issues.apache.org/jira/browse/HBASE-7441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-7441: --- Status: Open (was: Patch Available) Make ClusterManager in IntegrationTestingUtility pluggable -- Key: HBASE-7441 URL: https://issues.apache.org/jira/browse/HBASE-7441 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.3 Reporter: Liu Shaohui Priority: Minor Labels: patch Fix For: 0.94.4 Attachments: HBASE-7441-0.94-v1.patch Original Estimate: 3h Remaining Estimate: 3h After the patch HBASE-7009, we can use ChaosMonkey to test the Hbase cluster. The ClusterManager use ssh to stop/start the rs or master without passwd. To support other cluster manager tool, we need to make clusterManager in IntegrationTestingUtility pluggable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7441) Make ClusterManager in IntegrationTestingUtility pluggable
[ https://issues.apache.org/jira/browse/HBASE-7441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-7441: --- Labels: newbie patch (was: patch) Status: Patch Available (was: Open) Make ClusterManager in IntegrationTestingUtility pluggable -- Key: HBASE-7441 URL: https://issues.apache.org/jira/browse/HBASE-7441 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.3 Reporter: Liu Shaohui Priority: Minor Labels: patch, newbie Fix For: 0.94.4 Attachments: HBASE-7441-0.94-v1.patch Original Estimate: 3h Remaining Estimate: 3h After the patch HBASE-7009, we can use ChaosMonkey to test the Hbase cluster. The ClusterManager use ssh to stop/start the rs or master without passwd. To support other cluster manager tool, we need to make clusterManager in IntegrationTestingUtility pluggable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7441) Make ClusterManager in IntegrationTestingUtility pluggable
[ https://issues.apache.org/jira/browse/HBASE-7441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-7441: --- Fix Version/s: 0.94.4 Labels: patch (was: ) Status: Patch Available (was: Open) Make ClusterManager in IntegrationTestingUtility pluggable -- Key: HBASE-7441 URL: https://issues.apache.org/jira/browse/HBASE-7441 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.3 Reporter: Liu Shaohui Priority: Minor Labels: patch Fix For: 0.94.4 Attachments: HBASE-7441-0.94-v1.patch Original Estimate: 3h Remaining Estimate: 3h After the patch HBASE-7009, we can use ChaosMonkey to test the Hbase cluster. The ClusterManager use ssh to stop/start the rs or master without passwd. To support other cluster manager tool, we need to make clusterManager in IntegrationTestingUtility pluggable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7441) Make ClusterManager in IntegrationTestingUtility pluggable
[ https://issues.apache.org/jira/browse/HBASE-7441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-7441: --- Attachment: HBASE-7441-0.94-v1.patch Make ClusterManager in IntegrationTestingUtility pluggable -- Key: HBASE-7441 URL: https://issues.apache.org/jira/browse/HBASE-7441 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.3 Reporter: Liu Shaohui Priority: Minor Labels: patch Fix For: 0.94.4 Attachments: HBASE-7441-0.94-v1.patch Original Estimate: 3h Remaining Estimate: 3h After the patch HBASE-7009, we can use ChaosMonkey to test the Hbase cluster. The ClusterManager use ssh to stop/start the rs or master without passwd. To support other cluster manager tool, we need to make clusterManager in IntegrationTestingUtility pluggable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7441) Make ClusterManager in IntegrationTestingUtility pluggable
[ https://issues.apache.org/jira/browse/HBASE-7441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539905#comment-13539905 ] Liu Shaohui commented on HBASE-7441: Could you give some suggestion? [~sershe] Make ClusterManager in IntegrationTestingUtility pluggable -- Key: HBASE-7441 URL: https://issues.apache.org/jira/browse/HBASE-7441 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.3 Reporter: Liu Shaohui Priority: Minor Labels: newbie, patch Fix For: 0.94.4 Attachments: HBASE-7441-0.94-v1.patch Original Estimate: 3h Remaining Estimate: 3h After the patch HBASE-7009, we can use ChaosMonkey to test the Hbase cluster. The ClusterManager use ssh to stop/start the rs or master without passwd. To support other cluster manager tool, we need to make clusterManager in IntegrationTestingUtility pluggable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7440) ReplicationZookeeper#addPeer is racy
[ https://issues.apache.org/jira/browse/HBASE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539912#comment-13539912 ] Matteo Bertozzi commented on HBASE-7440: If you use createAndWatch() the node is not watched if it already exists, instead if you use createNodeIfNotExistsAndWatch(), as suggested by Lars, you get the watcher on the newly created one or on the old one. I think that you want the second behaviour, right? What do you expect if the node already exists, and is set to DISABLED? I think is fine in that state, right? so, you don't need to check it or force it to ENABLED. ReplicationZookeeper#addPeer is racy Key: HBASE-7440 URL: https://issues.apache.org/jira/browse/HBASE-7440 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 0.94.3 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.96.0, 0.94.4 Attachments: HBASE-7440-v0.patch, HBASE-7440-v1.patch While adding a peer, ReplicationZK does the znodes creation in three transactions. Create : a) peers znode b) peerId specific znode, and c) peerState znode There is a PeerWatcher which invokes getPeer() (after steps b) and c)). If it happens that while adding a peer, the control flows to getPeer() and step c) has not been processed, it may results in a state where the peer will not be added. This happens while running TestMasterReplication#testCyclicReplication(). {code} 2012-12-26 07:36:35,187 INFO [RegionServer:0;p0120.X,38423,1356536179470-EventThread] zookeeper.RecoverableZooKeeper(447): Node /2/replication/peers/1/peer-state already exists and this is not a retry 2012-12-26 07:36:35,188 ERROR [RegionServer:0;p0120.X,38423,1356536179470-EventThread] regionserver.ReplicationSourceManager$PeersWatcher(527): Error while adding a new peer org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /2/replication/peers/1/peer-state at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:428) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:410) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndWatch(ZKUtil.java:1044) at org.apache.hadoop.hbase.replication.ReplicationPeer.startStateTracker(ReplicationPeer.java:82) at org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:344) at org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:307) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$PeersWatcher.nodeChildrenChanged(ReplicationSourceManager.java:519) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:315) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) 2012-12-26 07:36:35,188 DEBUG [RegionServer:0;p0120.X,55742,1356536171947-EventThread] zookeeper.ZKUtil(1545): regionserver:55742-0x13bd7db39580004 Retrieved 36 byte(s) of data from znode /1/hbaseid; data=9ce66123-d3e8-4ae9-a249-afe03... {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7294) Check for snapshot file cleaners on start
[ https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7294: --- Attachment: HBASE-7294-v6.patch * Changed exception message using hbase.snapshot.enabled instead of the cleaners * Added warn on cleaners enabled, but snapshot.enabled false/not set * Added a test to verify different conf settings and expected result Check for snapshot file cleaners on start - Key: HBASE-7294 URL: https://issues.apache.org/jira/browse/HBASE-7294 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Affects Versions: hbase-6055 Reporter: Jesse Yates Assignee: Matteo Bertozzi Fix For: hbase-6055, 0.96.0 Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch, HBASE-7294-v3.patch, HBASE-7294-v4.patch, HBASE-7294-v5.patch, HBASE-7294-v6.patch Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to ensure that any hfiles or hlogs (respectively) that are currently part of a snapshot are not removed from their respective archive directories (.archive and .oldlogs). From Matteo Bertozzi: {quote} currently the snapshot cleaner is not in hbase-default.xml and there's no warning/exception on snapshot/restore operation, if not enabled. even if we add the cleaner to the hbase-default.xml how do we ensure that the user doesn't remove it? Do we want to hardcode the cleaner at master startup? Do we want to add a check in snapshot/restore that throws an exception if the cleaner is not enabled? {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7419) revisit hfilelink file name format.
[ https://issues.apache.org/jira/browse/HBASE-7419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539954#comment-13539954 ] Matteo Bertozzi commented on HBASE-7419: I agree, the introduction of a special character simplifies the identification of a HFileLink, and allows to remove the FileNotFoundException handling from the StoreFile. revisit hfilelink file name format. --- Key: HBASE-7419 URL: https://issues.apache.org/jira/browse/HBASE-7419 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Reporter: Jonathan Hsieh Fix For: hbase-6055, 0.96.0 Valid table names are concatted with a '.' to a valid regions names is also a valid table name, and lead to the incorrect interpretation. {code} true hfile name constraints: [0-9]+(?:_SeqID_[0-9]+)? region name constraints: [a-f0-9]{16} (but we currently just use [a-f0-9]+.) table name constraints : [a-zA-Z0-9_][a-zA-Z0-9_.-]* {code} Notice that the table name constraints completely covers all region name constraints and true hfile name constraints. (a valid hfile name is a valid part of a table name, and a valid enc region name is a valid part of a table name. Currently the hfilelink filename convention is hfile-region-table. Unfortunately, making a ref to this uses the name hfile-region-table.parentregion -- the contactnation of table.parentregion is a valid table name used to get interpreted as such. The fix in HBASE-7339 requires a FileNotFoundException before going down the hfile link resolution path. Regardless of what we do, we need to add some char invalid for table names to the hfilelink or reference filename convention. Suggestion: if we changed the order of the hfile-link name we could avoid some of the confusion -- table@region-hfile.parentregion (or some other separator char than '@') could be used to avoid handling on the initial filenotfoundexception but I think we'd still need a good chunk of the logic to handle opening half-storefile reader throw a hfilelink. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7400) ExportSnapshot mapper closes the FileSystem
[ https://issues.apache.org/jira/browse/HBASE-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539963#comment-13539963 ] Matteo Bertozzi commented on HBASE-7400: changes between hadoop 1 and 2. and since in the test, the mappers seems running in the same jvm fs.close() called by the mapper interferes with the one used by the main(). ExportSnapshot mapper closes the FileSystem --- Key: HBASE-7400 URL: https://issues.apache.org/jira/browse/HBASE-7400 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Blocker Fix For: hbase-6055, 0.96.0 Attachments: HBASE-7400-v0.patch The ExportSnapshotMapper closes the FileSystem on cleanup, and in case of shared instead, since there's no ref count, the job fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7188) Move classes into hbase-client
[ https://issues.apache.org/jira/browse/HBASE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539962#comment-13539962 ] Elliott Clark commented on HBASE-7188: -- [~stack] It seems like that's been changed for quite a while (Here's the file before we started splitting this out.): https://github.com/apache/hbase/blob/fd7647b25d316fef32413b3caeddcf6ba32485f9/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/ProtobufRpcEngine.java#L112 From a quick skim it looks like HBASE-5705 was the jira that changed that code. Move classes into hbase-client -- Key: HBASE-7188 URL: https://issues.apache.org/jira/browse/HBASE-7188 Project: HBase Issue Type: Sub-task Components: Client, IPC/RPC Affects Versions: 0.96.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.96.0 Attachments: HBASE-7188-0.patch, HBASE-7188-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539977#comment-13539977 ] Ted Yu commented on HBASE-5776: --- Integrated to trunk. Thanks for the patch, Liyin and lijin. Thanks for the review, Ram. HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: binlijin Fix For: 0.96.0 Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch, 5776-trunk-V5.patch, 5776-trunk-V6.patch, 5776-trunk-V7.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, HBASE-5776-trunk-V2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7400) ExportSnapshot mapper closes the FileSystem
[ https://issues.apache.org/jira/browse/HBASE-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13539979#comment-13539979 ] Ted Yu commented on HBASE-7400: --- I see. What do you think about my other comment w.r.t. 'new Configuration()' ? Thanks ExportSnapshot mapper closes the FileSystem --- Key: HBASE-7400 URL: https://issues.apache.org/jira/browse/HBASE-7400 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Blocker Fix For: hbase-6055, 0.96.0 Attachments: HBASE-7400-v0.patch The ExportSnapshotMapper closes the FileSystem on cleanup, and in case of shared instead, since there's no ref count, the job fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7428) There are multiple HConstants for configuring Zookeeper timeout
[ https://issues.apache.org/jira/browse/HBASE-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7428: Status: Open (was: Patch Available) There are multiple HConstants for configuring Zookeeper timeout --- Key: HBASE-7428 URL: https://issues.apache.org/jira/browse/HBASE-7428 Project: HBase Issue Type: Improvement Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Minor Labels: noob Fix For: 0.96.0 Attachments: 7428-consolidate-ZK_SESSION_TIMEOUT.0.diff From [~te...@apache.org] to dev@: {quote} There are two constants with the same value: HConstants.ZOOKEEPER_SESSION_TIMEOUT and HConstants.ZK_SESSION_TIMEOUT HConstants.ZOOKEEPER_SESSION_TIMEOUT is only used in tests. HConstants.ZK_SESSION_TIMEOUT is used by ZKUtil Shall we remove HConstants.ZOOKEEPER_SESSION_TIMEOUT and let tests use HConstants.ZK_SESSION_TIMEOUT ? {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7428) There are multiple HConstants for configuring Zookeeper timeout
[ https://issues.apache.org/jira/browse/HBASE-7428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-7428: Status: Patch Available (was: Open) There are multiple HConstants for configuring Zookeeper timeout --- Key: HBASE-7428 URL: https://issues.apache.org/jira/browse/HBASE-7428 Project: HBase Issue Type: Improvement Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Minor Labels: noob Fix For: 0.96.0 Attachments: 7428-consolidate-ZK_SESSION_TIMEOUT.0.diff From [~te...@apache.org] to dev@: {quote} There are two constants with the same value: HConstants.ZOOKEEPER_SESSION_TIMEOUT and HConstants.ZK_SESSION_TIMEOUT HConstants.ZOOKEEPER_SESSION_TIMEOUT is only used in tests. HConstants.ZK_SESSION_TIMEOUT is used by ZKUtil Shall we remove HConstants.ZOOKEEPER_SESSION_TIMEOUT and let tests use HConstants.ZK_SESSION_TIMEOUT ? {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file
[ https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-5598: --- Attachment: 5598.part2.patch Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file -- Key: HBASE-5598 URL: https://issues.apache.org/jira/browse/HBASE-5598 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.92.1, 0.94.0, 0.96.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Priority: Critical Fix For: 0.96.0 Attachments: 5598.part1.patch, 5598.part2.patch, ASF.LICENSE.NOT.GRANTED--findbugs-gui-report.jpg, HBASE-5598.patch There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to up the OK count. This may lead to other issues when we re-factor the code, if we induce new valid ones and remove invalid bugs also can not be reported by QA. So, I would propose to add the exclude filter file for findbugs(for the invalid bugs). If we find any valid ones, we can fix under this JIRA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file
[ https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1354#comment-1354 ] nkeywal commented on HBASE-5598: part2: - adds the dependency to findbugs for the annotations. It's a compile only dependency. License is LGPL. - suppress a few not critical warnings - fixes a few others Locally, I'm now at 144. Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file -- Key: HBASE-5598 URL: https://issues.apache.org/jira/browse/HBASE-5598 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.92.1, 0.94.0, 0.96.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Priority: Critical Fix For: 0.96.0 Attachments: 5598.part1.patch, 5598.part2.patch, 5598.part2.patch, ASF.LICENSE.NOT.GRANTED--findbugs-gui-report.jpg, HBASE-5598.patch There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to up the OK count. This may lead to other issues when we re-factor the code, if we induce new valid ones and remove invalid bugs also can not be reported by QA. So, I would propose to add the exclude filter file for findbugs(for the invalid bugs). If we find any valid ones, we can fix under this JIRA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file
[ https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-5598: --- Status: Open (was: Patch Available) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file -- Key: HBASE-5598 URL: https://issues.apache.org/jira/browse/HBASE-5598 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.94.0, 0.92.1, 0.96.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Priority: Critical Fix For: 0.96.0 Attachments: 5598.part1.patch, 5598.part2.patch, 5598.part2.patch, ASF.LICENSE.NOT.GRANTED--findbugs-gui-report.jpg, HBASE-5598.patch There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to up the OK count. This may lead to other issues when we re-factor the code, if we induce new valid ones and remove invalid bugs also can not be reported by QA. So, I would propose to add the exclude filter file for findbugs(for the invalid bugs). If we find any valid ones, we can fix under this JIRA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file
[ https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-5598: --- Attachment: 5598.part2.patch Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file -- Key: HBASE-5598 URL: https://issues.apache.org/jira/browse/HBASE-5598 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.92.1, 0.94.0, 0.96.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Priority: Critical Fix For: 0.96.0 Attachments: 5598.part1.patch, 5598.part2.patch, 5598.part2.patch, ASF.LICENSE.NOT.GRANTED--findbugs-gui-report.jpg, HBASE-5598.patch There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to up the OK count. This may lead to other issues when we re-factor the code, if we induce new valid ones and remove invalid bugs also can not be reported by QA. So, I would propose to add the exclude filter file for findbugs(for the invalid bugs). If we find any valid ones, we can fix under this JIRA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7442) HBase remote CopyTable not working when security enabled
James Kinley created HBASE-7442: --- Summary: HBase remote CopyTable not working when security enabled Key: HBASE-7442 URL: https://issues.apache.org/jira/browse/HBASE-7442 Project: HBase Issue Type: Bug Components: IPC/RPC, mapreduce, security Affects Versions: 0.92.1 Reporter: James Kinley When security is enabled, HBase CopyTable fails with Kerberos exception: {code} FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] {code} This is only when copying to remote HBase cluster (using either MRv1 or YARN), local copy works fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file
[ https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-5598: --- Status: Patch Available (was: Open) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file -- Key: HBASE-5598 URL: https://issues.apache.org/jira/browse/HBASE-5598 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.94.0, 0.92.1, 0.96.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Priority: Critical Fix For: 0.96.0 Attachments: 5598.part1.patch, 5598.part2.patch, 5598.part2.patch, ASF.LICENSE.NOT.GRANTED--findbugs-gui-report.jpg, HBASE-5598.patch There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to up the OK count. This may lead to other issues when we re-factor the code, if we induce new valid ones and remove invalid bugs also can not be reported by QA. So, I would propose to add the exclude filter file for findbugs(for the invalid bugs). If we find any valid ones, we can fix under this JIRA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6752) On region server failure, serve writes and timeranged reads during the log split
[ https://issues.apache.org/jira/browse/HBASE-6752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540005#comment-13540005 ] nkeywal commented on HBASE-6752: @Gregory: As you have unassigned the jira, I will have a look in the next weeks. Have you studied some options in more details and rejected them? On region server failure, serve writes and timeranged reads during the log split Key: HBASE-6752 URL: https://issues.apache.org/jira/browse/HBASE-6752 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.96.0 Reporter: nkeywal Priority: Minor Opening for write on failure would mean: - Assign the region to a new regionserver. It marks the region as recovering -- specific exception returned to the client when we cannot server. -- allow them to know where they stand. The exception can include some time information (failure stated on: ...) -- allow them to go immediately on the right regionserver, instead of retrying or calling the region holding meta to get the new address = save network calls, lower the load on meta. - Do the split as today. Priority is given to region server holding the new regions -- help to share the load balancing code: the split is done by region server considered as available for new regions -- help locality (the recovered edits are available on the region server) = lower the network usage - When the split is finished, we're done as of today - while the split is progressing, the region server can -- serve writes --- that's useful for all application that need to write but not read immediately: --- whatever logs events to analyze them later --- opentsdb is a perfect example. -- serve reads if they have a compatible time range. For heavily used tables, it could be an help, because: --- we can expect to have a few minutes of data only (as it's loaded) --- the heaviest queries, often accepts a few -or more- minutes delay. Some What if: 1) the split fails = Retry until it works. As today. Just that we serves writes. We need to know (as today) that the region has not recovered if we fail again. 2) the regionserver fails during the split = As 1 and as of today/ 3) the regionserver fails after the split but before the state change to fully available. = New assign. More logs to split (the ones already dones and the new ones). 4) the assignment fails = Retry until it works. As today. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7442) HBase remote CopyTable not working when security enabled
[ https://issues.apache.org/jira/browse/HBASE-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540007#comment-13540007 ] James Kinley commented on HBASE-7442: - To give some context for the attached log ({{attempt_201212271546_0001_m_00_0.log}}). I have setup 2 clusters: Cluster 1 has two nodes running the following services: one.kinley.com: HDFS, MRv1 two.kinley.com: HBase, ZK Cluster 2 has a single node running the following: three.kinley.com: HDFS, HBase, ZK Security is enabled on both clusters and all principals are in the same realm (so cross-realm support is not required). I'm trying to copy an HBase table from cluster 1 to cluster 2: {code} two.kinley.com:~$ hbase org.apache.hadoop.hbase.mapreduce.CopyTable -Dmapred.child.java.opts=-Dsun.security.krb5.debug=true -Dmapred.map.child.log.level=DEBUG --peer.adr=three.kinley.com:2181:/hbase --new.name=Converged-headers Converged-headers {code} Running a local2local CopyTable on cluster 1 and 2 works ok. HBase remote CopyTable not working when security enabled Key: HBASE-7442 URL: https://issues.apache.org/jira/browse/HBASE-7442 Project: HBase Issue Type: Bug Components: IPC/RPC, mapreduce, security Affects Versions: 0.92.1 Reporter: James Kinley When security is enabled, HBase CopyTable fails with Kerberos exception: {code} FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] {code} This is only when copying to remote HBase cluster (using either MRv1 or YARN), local copy works fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7442) HBase remote CopyTable not working when security enabled
[ https://issues.apache.org/jira/browse/HBASE-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Kinley updated HBASE-7442: Attachment: attempt_201212271546_0001_m_00_0.log HBase remote CopyTable not working when security enabled Key: HBASE-7442 URL: https://issues.apache.org/jira/browse/HBASE-7442 Project: HBase Issue Type: Bug Components: IPC/RPC, mapreduce, security Affects Versions: 0.92.1 Reporter: James Kinley Attachments: attempt_201212271546_0001_m_00_0.log When security is enabled, HBase CopyTable fails with Kerberos exception: {code} FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] {code} This is only when copying to remote HBase cluster (using either MRv1 or YARN), local copy works fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file
[ https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540027#comment-13540027 ] stack commented on HBASE-5598: -- @nkeywal Do you want to do part2 in another issue linked to from here since we have already committed a patch against this issue? Why move the defines in HRegionInfo? This is a nice trick: {code} +@edu.umd.cs.findbugs.annotations.SuppressWarnings (value=ES_COMPARING_STRINGS_WITH_EQ, +justification=Optimization) {code} What warning do we suppress by doing this: - return new StringBuilder(AuthResult) - .append(toContextString()).toString(); + return AuthResult + toContextString(); I know its not 'important' but these kinda findings are good: - StringBuffer sb = new StringBuffer(); + StringBuilder sb = new StringBuilder(); We are going back to findbugs 2.0.1 from 2.5.2? Is this define used anywhere: findbugs-maven-plugin.version Good work Nicolas Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file -- Key: HBASE-5598 URL: https://issues.apache.org/jira/browse/HBASE-5598 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.92.1, 0.94.0, 0.96.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Priority: Critical Fix For: 0.96.0 Attachments: 5598.part1.patch, 5598.part2.patch, 5598.part2.patch, ASF.LICENSE.NOT.GRANTED--findbugs-gui-report.jpg, HBASE-5598.patch There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to up the OK count. This may lead to other issues when we re-factor the code, if we induce new valid ones and remove invalid bugs also can not be reported by QA. So, I would propose to add the exclude filter file for findbugs(for the invalid bugs). If we find any valid ones, we can fix under this JIRA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7442) HBase remote CopyTable not working when security enabled
[ https://issues.apache.org/jira/browse/HBASE-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540031#comment-13540031 ] James Kinley commented on HBASE-7442: - I noticed that *{{CopyTable#createSubmittableJob}}*, and specifically *{{TableMapReduceUtil#initCredentials}}* was only acquiring a delegation token for the local HBase cluster so I added the following to *{{TableMapReduceUtil#initCredentials}}*: {code} String quorumAddress = job.getConfiguration().get(TableOutputFormat.QUORUM_ADDRESS); if (quorumAddress != null) { String[] parts = ZKUtil.transformClusterKey(quorumAddress); Configuration peerConf = HBaseConfiguration.create(job.getConfiguration()); peerConf.set(HConstants.ZOOKEEPER_QUORUM, parts[0]); peerConf.set(hbase.zookeeper.client.port, parts[1]); peerConf.set(HConstants.ZOOKEEPER_ZNODE_PARENT, parts[2]); User.getCurrent().obtainAuthTokenForJob(peerConf, job); } {code} The user now has the correct tokens: Cluster 1 ID = 7bb01c84-3c66-49ca-82d2-2f9218f13dad Cluster 2 ID = d4653a37-83df-4929-b950-e03987c3b4e8 {code} INFO token.TokenUtil: Obtained token HBASE_AUTH_TOKEN for user kinley/ad...@kinley.com on cluster 7bb01c84-3c66-49ca-82d2-2f9218f13dad INFO token.TokenUtil: Obtained token HBASE_AUTH_TOKEN for user kinley/ad...@kinley.com on cluster d4653a37-83df-4929-b950-e03987c3b4e8 {code} But when initialising the *{{HTable}}* connection in the mapper, the *{{TokenSelector}}* in *{{SecureClient#SecureConnection}}* returns the wrong token from the user's token cache. This is because the token cache is keyed on {{clusterId}}, and despite the token for the local cluster existing in the cache ({{key=7bb01c84-3c66-49ca-82d2-2f9218f13dad}}), the {{clusterId}} is fixed at this point in *{{HBaseClient}}* to the remote cluster's ID ({{key=d4653a37-83df-4929-b950-e03987c3b4e8}}), and it is this ID that is always passed to the *{{TokenSelector}}*. I can think of a couple of possible solutions: 1. Either pass the {{clusterId}} into *{{SecureClient#SecureConnection}}* as part of the *{{ConnectionId}}*. This way it can be used by the *{{TokenSelector}}* instead of the fixed ID in the parent class *{{HBaseClient}}*?, or 2. *{{SecureRpcEngine}}* caches multiple clients based on a *{{SocketFactory}}*, so we could maintain separate clients for both the local and remote clusters here? HBase remote CopyTable not working when security enabled Key: HBASE-7442 URL: https://issues.apache.org/jira/browse/HBASE-7442 Project: HBase Issue Type: Bug Components: IPC/RPC, mapreduce, security Affects Versions: 0.92.1 Reporter: James Kinley Attachments: attempt_201212271546_0001_m_00_0.log When security is enabled, HBase CopyTable fails with Kerberos exception: {code} FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] {code} This is only when copying to remote HBase cluster (using either MRv1 or YARN), local copy works fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7400) ExportSnapshot mapper closes the FileSystem
[ https://issues.apache.org/jira/browse/HBASE-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7400: --- Attachment: HBASE-7400-v1.patch sorry Ted, I tend to forget to reply, to comments/nits/suggestions that I agree with, and I'm going change the code. Anyway, you're right. The configuration doesn't matter since the user should specify a qualified path hdfs://cluster2 and in case of unqualified path, you've an unexpected result in both case. but at least using the getConf() you're sure that you get all your config (HBaseConfig is used) and maybe your fs.default.name is inside one of those. ExportSnapshot mapper closes the FileSystem --- Key: HBASE-7400 URL: https://issues.apache.org/jira/browse/HBASE-7400 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Blocker Fix For: hbase-6055, 0.96.0 Attachments: HBASE-7400-v0.patch, HBASE-7400-v1.patch The ExportSnapshotMapper closes the FileSystem on cleanup, and in case of shared instead, since there's no ref count, the job fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7400) ExportSnapshot mapper closes the FileSystem
[ https://issues.apache.org/jira/browse/HBASE-7400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540039#comment-13540039 ] Ted Yu commented on HBASE-7400: --- Thanks for the quick response. +1 on second patch. ExportSnapshot mapper closes the FileSystem --- Key: HBASE-7400 URL: https://issues.apache.org/jira/browse/HBASE-7400 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Blocker Fix For: hbase-6055, 0.96.0 Attachments: HBASE-7400-v0.patch, HBASE-7400-v1.patch The ExportSnapshotMapper closes the FileSystem on cleanup, and in case of shared instead, since there's no ref count, the job fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540047#comment-13540047 ] Hudson commented on HBASE-5776: --- Integrated in HBase-TRUNK #3660 (See [https://builds.apache.org/job/HBase-TRUNK/3660/]) HBASE-5776 HTableMultiplexer (Liyin, binlijin and Ted Yu) (Revision 1426221) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: binlijin Fix For: 0.96.0 Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch, 5776-trunk-V5.patch, 5776-trunk-V6.patch, 5776-trunk-V7.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, HBASE-5776-trunk-V2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7442) HBase remote CopyTable not working when security enabled
[ https://issues.apache.org/jira/browse/HBASE-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7442: - Fix Version/s: 0.94.5 0.96.0 HBase remote CopyTable not working when security enabled Key: HBASE-7442 URL: https://issues.apache.org/jira/browse/HBASE-7442 Project: HBase Issue Type: Bug Components: IPC/RPC, mapreduce, security Affects Versions: 0.92.1 Reporter: James Kinley Fix For: 0.96.0, 0.94.5 Attachments: attempt_201212271546_0001_m_00_0.log When security is enabled, HBase CopyTable fails with Kerberos exception: {code} FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] {code} This is only when copying to remote HBase cluster (using either MRv1 or YARN), local copy works fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7442) HBase remote CopyTable not working when security enabled
[ https://issues.apache.org/jira/browse/HBASE-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540051#comment-13540051 ] Lars Hofhansl commented on HBASE-7442: -- Thanks James. [~apurtell], [~ghelmling] any thoughts? HBase remote CopyTable not working when security enabled Key: HBASE-7442 URL: https://issues.apache.org/jira/browse/HBASE-7442 Project: HBase Issue Type: Bug Components: IPC/RPC, mapreduce, security Affects Versions: 0.92.1 Reporter: James Kinley Fix For: 0.96.0, 0.94.5 Attachments: attempt_201212271546_0001_m_00_0.log When security is enabled, HBase CopyTable fails with Kerberos exception: {code} FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] {code} This is only when copying to remote HBase cluster (using either MRv1 or YARN), local copy works fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file
[ https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540050#comment-13540050 ] nkeywal commented on HBASE-5598: bq. @nkeywal Do you want to do part2 in another issue linked to from here since we have already committed a patch against this issue? Ok, I will do that. bq. Why move the defines in HRegionInfo? We're constructing an object while the static fields are not yet all initialized: {code} class A{ static A = new A(); static int i = 15; A(){ print i= + i; } } {code} prints i=0; Tricky isn't it? bq. return new StringBuilder(AuthResult) .append(toContextString()).toString(); + return AuthResult + toContextString(); Don't remember its name (it could be an IntelliJ warning as well, I fix both), but it's just that the new code is simpler and at least as efficient as the previous one, as AuthResult is a constant. bq. We are going back to findbugs 2.0.1 from 2.5.2? There are two versions, one for findbugs, one for the maven plugin. Maven plugin 2.5.2 includes findbugs 2.0.1. The naming was previously confusing, so I renamed the variables. And we need to explicitly define the two versions, as we're including the plugin but as well the annotations. Last findbugs is 2.0.2, but I put 2.0.1 to match the version included by the maven plugin bq. Is this define used anywhere: findbugs-maven-plugin.version Yep, in the main pom, when including the maven plugin. Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file -- Key: HBASE-5598 URL: https://issues.apache.org/jira/browse/HBASE-5598 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.92.1, 0.94.0, 0.96.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Priority: Critical Fix For: 0.96.0 Attachments: 5598.part1.patch, 5598.part2.patch, 5598.part2.patch, ASF.LICENSE.NOT.GRANTED--findbugs-gui-report.jpg, HBASE-5598.patch There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to up the OK count. This may lead to other issues when we re-factor the code, if we induce new valid ones and remove invalid bugs also can not be reported by QA. So, I would propose to add the exclude filter file for findbugs(for the invalid bugs). If we find any valid ones, we can fix under this JIRA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7443) More findbugs fixes
nkeywal created HBASE-7443: -- Summary: More findbugs fixes Key: HBASE-7443 URL: https://issues.apache.org/jira/browse/HBASE-7443 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 0.96.0 Attachments: 7443.patch It - adds the dependency to findbugs for the annotations. It's a compile only dependency. License is LGPL. - suppresses a few not critical warnings - fixes a few others Locally, I'm now at 144. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.
[ https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540054#comment-13540054 ] Ted Yu commented on HBASE-5416: --- bq. Now in this case i will have only the joinedScanner heap. I think you meant we only have storeHeap. Here is related code from HRegion.nextInternal() of 0.94 branch: {code} if (isStopRow(currentRow, offset, length)) { if (filter != null filter.hasFilterRow()) { filter.filterRow(results); } if (filter != null filter.filterRow()) { results.clear(); } return false; {code} We can see the logic is the same as that in patch v12. Improve performance of scans with some kind of filters. --- Key: HBASE-5416 URL: https://issues.apache.org/jira/browse/HBASE-5416 Project: HBase Issue Type: Improvement Components: Filters, Performance, regionserver Affects Versions: 0.90.4 Reporter: Max Lapan Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: 5416-Filtered_scans_v6.patch, 5416-v5.txt, 5416-v6.txt, Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch, Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch, Filtered_scans_v7.patch, HBASE-5416-v10.patch, HBASE-5416-v11.patch, HBASE-5416-v12.patch, HBASE-5416-v12.patch, HBASE-5416-v7-rebased.patch, HBASE-5416-v8.patch, HBASE-5416-v9.patch When the scan is performed, whole row is loaded into result list, after that filter (if exists) is applied to detect that row is needed. But when scan is performed on several CFs and filter checks only data from the subset of these CFs, data from CFs, not checked by a filter is not needed on a filter stage. Only when we decided to include current row. And in such case we can significantly reduce amount of IO performed by a scan, by loading only values, actually checked by a filter. For example, we have two CFs: flags and snap. Flags is quite small (bunch of megabytes) and is used to filter large entries from snap. Snap is very large (10s of GB) and it is quite costly to scan it. If we needed only rows with some flag specified, we use SingleColumnValueFilter to limit result to only small subset of region. But current implementation is loading both CFs to perform scan, when only small subset is needed. Attached patch adds one routine to Filter interface to allow filter to specify which CF is needed to it's operation. In HRegion, we separate all scanners into two groups: needed for filter and the rest (joined). When new row is considered, only needed data is loaded, filter applied, and only if filter accepts the row, rest of data is loaded. At our data, this speeds up such kind of scans 30-50 times. Also, this gives us the way to better normalize the data into separate columns by optimizing the scans performed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7443) More findbugs fixes
[ https://issues.apache.org/jira/browse/HBASE-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-7443: --- Attachment: 7443.patch More findbugs fixes --- Key: HBASE-7443 URL: https://issues.apache.org/jira/browse/HBASE-7443 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 0.96.0 Attachments: 7443.patch It - adds the dependency to findbugs for the annotations. It's a compile only dependency. License is LGPL. - suppresses a few not critical warnings - fixes a few others Locally, I'm now at 144. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5416) Improve performance of scans with some kind of filters.
[ https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-5416: -- Attachment: 5416-v13.patch Patch v13 rebases on trunk. TestSingleColumnValueExcludeFilter, TestHRegion and TestJoinedScanners pass. Improve performance of scans with some kind of filters. --- Key: HBASE-5416 URL: https://issues.apache.org/jira/browse/HBASE-5416 Project: HBase Issue Type: Improvement Components: Filters, Performance, regionserver Affects Versions: 0.90.4 Reporter: Max Lapan Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: 5416-Filtered_scans_v6.patch, 5416-v13.patch, 5416-v5.txt, 5416-v6.txt, Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch, Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch, Filtered_scans_v7.patch, HBASE-5416-v10.patch, HBASE-5416-v11.patch, HBASE-5416-v12.patch, HBASE-5416-v12.patch, HBASE-5416-v7-rebased.patch, HBASE-5416-v8.patch, HBASE-5416-v9.patch When the scan is performed, whole row is loaded into result list, after that filter (if exists) is applied to detect that row is needed. But when scan is performed on several CFs and filter checks only data from the subset of these CFs, data from CFs, not checked by a filter is not needed on a filter stage. Only when we decided to include current row. And in such case we can significantly reduce amount of IO performed by a scan, by loading only values, actually checked by a filter. For example, we have two CFs: flags and snap. Flags is quite small (bunch of megabytes) and is used to filter large entries from snap. Snap is very large (10s of GB) and it is quite costly to scan it. If we needed only rows with some flag specified, we use SingleColumnValueFilter to limit result to only small subset of region. But current implementation is loading both CFs to perform scan, when only small subset is needed. Attached patch adds one routine to Filter interface to allow filter to specify which CF is needed to it's operation. In HRegion, we separate all scanners into two groups: needed for filter and the rest (joined). When new row is considered, only needed data is loaded, filter applied, and only if filter accepts the row, rest of data is loaded. At our data, this speeds up such kind of scans 30-50 times. Also, this gives us the way to better normalize the data into separate columns by optimizing the scans performed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.
[ https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540062#comment-13540062 ] Max Lapan commented on HBASE-5416: -- bq. I think you meant we only have storeHeap. No, exactly one KVS in joinedScanner heap and empty storeHeap. It was caused by {{!scan.doLoadColumnFamiliesOnDemand()}} extra condition in constructor. Improve performance of scans with some kind of filters. --- Key: HBASE-5416 URL: https://issues.apache.org/jira/browse/HBASE-5416 Project: HBase Issue Type: Improvement Components: Filters, Performance, regionserver Affects Versions: 0.90.4 Reporter: Max Lapan Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: 5416-Filtered_scans_v6.patch, 5416-v13.patch, 5416-v5.txt, 5416-v6.txt, Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch, Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch, Filtered_scans_v7.patch, HBASE-5416-v10.patch, HBASE-5416-v11.patch, HBASE-5416-v12.patch, HBASE-5416-v12.patch, HBASE-5416-v7-rebased.patch, HBASE-5416-v8.patch, HBASE-5416-v9.patch When the scan is performed, whole row is loaded into result list, after that filter (if exists) is applied to detect that row is needed. But when scan is performed on several CFs and filter checks only data from the subset of these CFs, data from CFs, not checked by a filter is not needed on a filter stage. Only when we decided to include current row. And in such case we can significantly reduce amount of IO performed by a scan, by loading only values, actually checked by a filter. For example, we have two CFs: flags and snap. Flags is quite small (bunch of megabytes) and is used to filter large entries from snap. Snap is very large (10s of GB) and it is quite costly to scan it. If we needed only rows with some flag specified, we use SingleColumnValueFilter to limit result to only small subset of region. But current implementation is loading both CFs to perform scan, when only small subset is needed. Attached patch adds one routine to Filter interface to allow filter to specify which CF is needed to it's operation. In HRegion, we separate all scanners into two groups: needed for filter and the rest (joined). When new row is considered, only needed data is loaded, filter applied, and only if filter accepts the row, rest of data is loaded. At our data, this speeds up such kind of scans 30-50 times. Also, this gives us the way to better normalize the data into separate columns by optimizing the scans performed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7390) Add extra test cases for assignement on the region server and fix the related issues
[ https://issues.apache.org/jira/browse/HBASE-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540064#comment-13540064 ] Jimmy Xiang commented on HBASE-7390: Here is the link to the review: https://reviews.apache.org/r/8754/ I will take a look today. Add extra test cases for assignement on the region server and fix the related issues Key: HBASE-7390 URL: https://issues.apache.org/jira/browse/HBASE-7390 Project: HBase Issue Type: Bug Components: Region Assignment, regionserver Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Fix For: 0.96.0 Attachments: 7390.v1.patch, 7390.v2.patch, 7390.v3.patch, 7390.v4.patch, 7390.v6.patch, 7390.v7.patch, 7390.v8.patch, assignment_zk_states.jpg We don't have a lot of tests on the region server itself. Here are some. Some of them are failing, feedback welcome. See as well the attached state diagram for the ZK nodes on assignment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7258) Hbase needs to create baseZNode recursivly
[ https://issues.apache.org/jira/browse/HBASE-7258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540068#comment-13540068 ] ramkrishna.s.vasudevan commented on HBASE-7258: --- @Stack/Ted I have given my word to Liu on commiting this. Pls let me know if we can commit this. Hbase needs to create baseZNode recursivly -- Key: HBASE-7258 URL: https://issues.apache.org/jira/browse/HBASE-7258 Project: HBase Issue Type: Bug Components: Zookeeper Affects Versions: 0.94.2 Reporter: Liu Shaohui Priority: Minor Labels: zookeeper, zookeeper.znode.parent Attachments: HBASE-7258-0.94.patch, HBASE-7258.diff, HBASE-7258.patch Original Estimate: 1h Remaining Estimate: 1h In deploy env, multi small hbase clusters may share a same zk cluster. So, for hbase cluster1, its parent znode is /hbase/cluster1. But in hbase version 0.94.1, hbase use ZKUtil.createAndFailSilent(this, baseZNode) to create parent path and it will throw a NoNode exception if znode /hbase donot exist. We want to change it to ZKUtil.createWithParents(this, baseZNode); to suport create baseZNode recursivly. The NoNode exception is: java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1792) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:146) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:77) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1806) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/cluster1 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:420) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:402) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:905) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:166) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:159) at org.apache.hadoop.hbase.master.HMaster.init(HMaster.java:282) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hbase.master.H -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.
[ https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540069#comment-13540069 ] ramkrishna.s.vasudevan commented on HBASE-5416: --- Yes Max that's what i meant. Ted, just take the condition where isFamilyEssential = true. Also scan says lazy loading of CF is allowed. (and also only one CF). So storeHeap will be null in this case. Improve performance of scans with some kind of filters. --- Key: HBASE-5416 URL: https://issues.apache.org/jira/browse/HBASE-5416 Project: HBase Issue Type: Improvement Components: Filters, Performance, regionserver Affects Versions: 0.90.4 Reporter: Max Lapan Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: 5416-Filtered_scans_v6.patch, 5416-v13.patch, 5416-v5.txt, 5416-v6.txt, Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch, Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch, Filtered_scans_v7.patch, HBASE-5416-v10.patch, HBASE-5416-v11.patch, HBASE-5416-v12.patch, HBASE-5416-v12.patch, HBASE-5416-v7-rebased.patch, HBASE-5416-v8.patch, HBASE-5416-v9.patch When the scan is performed, whole row is loaded into result list, after that filter (if exists) is applied to detect that row is needed. But when scan is performed on several CFs and filter checks only data from the subset of these CFs, data from CFs, not checked by a filter is not needed on a filter stage. Only when we decided to include current row. And in such case we can significantly reduce amount of IO performed by a scan, by loading only values, actually checked by a filter. For example, we have two CFs: flags and snap. Flags is quite small (bunch of megabytes) and is used to filter large entries from snap. Snap is very large (10s of GB) and it is quite costly to scan it. If we needed only rows with some flag specified, we use SingleColumnValueFilter to limit result to only small subset of region. But current implementation is loading both CFs to perform scan, when only small subset is needed. Attached patch adds one routine to Filter interface to allow filter to specify which CF is needed to it's operation. In HRegion, we separate all scanners into two groups: needed for filter and the rest (joined). When new row is considered, only needed data is loaded, filter applied, and only if filter accepts the row, rest of data is loaded. At our data, this speeds up such kind of scans 30-50 times. Also, this gives us the way to better normalize the data into separate columns by optimizing the scans performed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.
[ https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540070#comment-13540070 ] Ted Yu commented on HBASE-5416: --- Here is the change related to Max's comment above: {code} +if (this.filter == null || !scan.doLoadColumnFamiliesOnDemand() + || this.filter.isFamilyEssential(entry.getKey())) { + scanners.add(scanner); {code} Ram said: bq. Also scan says lazy loading of CF is allowed. So the condition {{!scan.doLoadColumnFamiliesOnDemand()}} should be false, right ? Improve performance of scans with some kind of filters. --- Key: HBASE-5416 URL: https://issues.apache.org/jira/browse/HBASE-5416 Project: HBase Issue Type: Improvement Components: Filters, Performance, regionserver Affects Versions: 0.90.4 Reporter: Max Lapan Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: 5416-Filtered_scans_v6.patch, 5416-v13.patch, 5416-v5.txt, 5416-v6.txt, Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch, Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch, Filtered_scans_v7.patch, HBASE-5416-v10.patch, HBASE-5416-v11.patch, HBASE-5416-v12.patch, HBASE-5416-v12.patch, HBASE-5416-v7-rebased.patch, HBASE-5416-v8.patch, HBASE-5416-v9.patch When the scan is performed, whole row is loaded into result list, after that filter (if exists) is applied to detect that row is needed. But when scan is performed on several CFs and filter checks only data from the subset of these CFs, data from CFs, not checked by a filter is not needed on a filter stage. Only when we decided to include current row. And in such case we can significantly reduce amount of IO performed by a scan, by loading only values, actually checked by a filter. For example, we have two CFs: flags and snap. Flags is quite small (bunch of megabytes) and is used to filter large entries from snap. Snap is very large (10s of GB) and it is quite costly to scan it. If we needed only rows with some flag specified, we use SingleColumnValueFilter to limit result to only small subset of region. But current implementation is loading both CFs to perform scan, when only small subset is needed. Attached patch adds one routine to Filter interface to allow filter to specify which CF is needed to it's operation. In HRegion, we separate all scanners into two groups: needed for filter and the rest (joined). When new row is considered, only needed data is loaded, filter applied, and only if filter accepts the row, rest of data is loaded. At our data, this speeds up such kind of scans 30-50 times. Also, this gives us the way to better normalize the data into separate columns by optimizing the scans performed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6752) On region server failure, serve writes and timeranged reads during the log split
[ https://issues.apache.org/jira/browse/HBASE-6752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540076#comment-13540076 ] Gregory Chanan commented on HBASE-6752: --- @nkeywal: didn't study anything in too much depth. For the read part, my thought was to implement a config (in HTableDescriptor?) that would reject user-set timestamps on writes, so we know for sure there can't be any writes in the timestamp range that need to be replayed from the WAL. I suspect there are other optimizations we could do with that information, but haven't thought it through. For writes, do you create a new WAL for the new writes that are happening while the log is still replaying? If so, management could be complicated and it might make sense to have support for multiple WALs already before tackling that. If not (you write to the same WAL), would that even work? I guess you would want to avoid replaying the new writes (might be okay if all WAL updates are idempotent, but could be an issue if a lot of writes go in during the replay time). On region server failure, serve writes and timeranged reads during the log split Key: HBASE-6752 URL: https://issues.apache.org/jira/browse/HBASE-6752 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.96.0 Reporter: nkeywal Priority: Minor Opening for write on failure would mean: - Assign the region to a new regionserver. It marks the region as recovering -- specific exception returned to the client when we cannot server. -- allow them to know where they stand. The exception can include some time information (failure stated on: ...) -- allow them to go immediately on the right regionserver, instead of retrying or calling the region holding meta to get the new address = save network calls, lower the load on meta. - Do the split as today. Priority is given to region server holding the new regions -- help to share the load balancing code: the split is done by region server considered as available for new regions -- help locality (the recovered edits are available on the region server) = lower the network usage - When the split is finished, we're done as of today - while the split is progressing, the region server can -- serve writes --- that's useful for all application that need to write but not read immediately: --- whatever logs events to analyze them later --- opentsdb is a perfect example. -- serve reads if they have a compatible time range. For heavily used tables, it could be an help, because: --- we can expect to have a few minutes of data only (as it's loaded) --- the heaviest queries, often accepts a few -or more- minutes delay. Some What if: 1) the split fails = Retry until it works. As today. Just that we serves writes. We need to know (as today) that the region has not recovered if we fail again. 2) the regionserver fails during the split = As 1 and as of today/ 3) the regionserver fails after the split but before the state change to fully available. = New assign. More logs to split (the ones already dones and the new ones). 4) the assignment fails = Retry until it works. As today. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7440) ReplicationZookeeper#addPeer is racy
[ https://issues.apache.org/jira/browse/HBASE-7440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-7440: --- Attachment: HBASE-7440-v2.patch Using ZKUtil.createNodeIfNotExistsAndWatch as suggested by Lars. Didn't notice this API earlier! Yes Matteo, the default value of znode is ENABLED, and this code is called while creating one. So, no check needed or such. ReplicationZookeeper#addPeer is racy Key: HBASE-7440 URL: https://issues.apache.org/jira/browse/HBASE-7440 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 0.94.3 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.96.0, 0.94.4 Attachments: HBASE-7440-v0.patch, HBASE-7440-v1.patch, HBASE-7440-v2.patch While adding a peer, ReplicationZK does the znodes creation in three transactions. Create : a) peers znode b) peerId specific znode, and c) peerState znode There is a PeerWatcher which invokes getPeer() (after steps b) and c)). If it happens that while adding a peer, the control flows to getPeer() and step c) has not been processed, it may results in a state where the peer will not be added. This happens while running TestMasterReplication#testCyclicReplication(). {code} 2012-12-26 07:36:35,187 INFO [RegionServer:0;p0120.X,38423,1356536179470-EventThread] zookeeper.RecoverableZooKeeper(447): Node /2/replication/peers/1/peer-state already exists and this is not a retry 2012-12-26 07:36:35,188 ERROR [RegionServer:0;p0120.X,38423,1356536179470-EventThread] regionserver.ReplicationSourceManager$PeersWatcher(527): Error while adding a new peer org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /2/replication/peers/1/peer-state at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:428) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:410) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndWatch(ZKUtil.java:1044) at org.apache.hadoop.hbase.replication.ReplicationPeer.startStateTracker(ReplicationPeer.java:82) at org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:344) at org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:307) at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$PeersWatcher.nodeChildrenChanged(ReplicationSourceManager.java:519) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:315) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) 2012-12-26 07:36:35,188 DEBUG [RegionServer:0;p0120.X,55742,1356536171947-EventThread] zookeeper.ZKUtil(1545): regionserver:55742-0x13bd7db39580004 Retrieved 36 byte(s) of data from znode /1/hbaseid; data=9ce66123-d3e8-4ae9-a249-afe03... {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.
[ https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540078#comment-13540078 ] Ted Yu commented on HBASE-5416: --- Here is code from patch v7: {code} +if (this.filter == null || this.filter.isFamilyEssential(entry.getKey())) { + scanners.add(scanner); {code} In the above case, I don't see difference between v7 and v12 w.r.t. populating scanners List. Improve performance of scans with some kind of filters. --- Key: HBASE-5416 URL: https://issues.apache.org/jira/browse/HBASE-5416 Project: HBase Issue Type: Improvement Components: Filters, Performance, regionserver Affects Versions: 0.90.4 Reporter: Max Lapan Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: 5416-Filtered_scans_v6.patch, 5416-v13.patch, 5416-v5.txt, 5416-v6.txt, Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch, Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch, Filtered_scans_v7.patch, HBASE-5416-v10.patch, HBASE-5416-v11.patch, HBASE-5416-v12.patch, HBASE-5416-v12.patch, HBASE-5416-v7-rebased.patch, HBASE-5416-v8.patch, HBASE-5416-v9.patch When the scan is performed, whole row is loaded into result list, after that filter (if exists) is applied to detect that row is needed. But when scan is performed on several CFs and filter checks only data from the subset of these CFs, data from CFs, not checked by a filter is not needed on a filter stage. Only when we decided to include current row. And in such case we can significantly reduce amount of IO performed by a scan, by loading only values, actually checked by a filter. For example, we have two CFs: flags and snap. Flags is quite small (bunch of megabytes) and is used to filter large entries from snap. Snap is very large (10s of GB) and it is quite costly to scan it. If we needed only rows with some flag specified, we use SingleColumnValueFilter to limit result to only small subset of region. But current implementation is loading both CFs to perform scan, when only small subset is needed. Attached patch adds one routine to Filter interface to allow filter to specify which CF is needed to it's operation. In HRegion, we separate all scanners into two groups: needed for filter and the rest (joined). When new row is considered, only needed data is loaded, filter applied, and only if filter accepts the row, rest of data is loaded. At our data, this speeds up such kind of scans 30-50 times. Also, this gives us the way to better normalize the data into separate columns by optimizing the scans performed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7263) Investigate more fine grained locking for checkAndPut/append/increment
[ https://issues.apache.org/jira/browse/HBASE-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540080#comment-13540080 ] Gregory Chanan commented on HBASE-7263: --- bq. Does your test just operate one rowkey? The full test description is given above. The relevant part to your question: - Each increment uniformly distributed over 500,000 rows bq. Do you mean it is not per-row for option 2( Have an MVCC per-row (table configuration): this avoids the unnecessary contention of 1)) Option 2 is per-row but that is not what is implemented today and this JIRA takes a different approach. Investigate more fine grained locking for checkAndPut/append/increment -- Key: HBASE-7263 URL: https://issues.apache.org/jira/browse/HBASE-7263 Project: HBase Issue Type: Improvement Components: Transactions/MVCC Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Minor HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut: {quote} 1) Waiting for the MVCC to advance for read/updates: the downside is that you have to wait for updates on other rows. 2) Have an MVCC per-row (table configuration): this avoids the unnecessary contention of 1) 3) Transform the read/updates to write-only with rollup on read.. E.g. an increment would just have the number of values to increment. {quote} HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned, is that you have to wait for updates on other rows, since MVCC is per-row. Another option occurred to me that I think is worth investigating: rely on a row-level read/write lock rather than MVCC. Here is pseudo-code for what exists today for read/updates like checkAndPut {code} (1) Acquire RowLock (1a) BeginMVCC + Finish MVCC (2) Begin MVCC (3) Do work (4) Release RowLock (5) Append to WAL (6) Finish MVCC {code} Write-only operations (e.g. puts) are the same, just without step 1a. Now, consider the following instead: {code} (1) Acquire RowLock (1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC) (1b) Grab RowReadLock (new step!) (2) Begin MVCC (3) Do work (4) Release RowLock (5) Append to WAL (6) Finish MVCC (7) Release RowReadLock (new step!) {code} As before, write-only operations are the same, just without step 1a. The difference here is that writes grab a row-level read lock and hold it until the MVCC is completed. The nice property that this gives you is that read/updates can tell when the MVCC is done on a per-row basis, because they can just try to acquire the write-lock which will block until the MVCC is competed for that row in step 7. There is overhead for acquiring the read lock that I need to measure, but it should be small, since there will never be any blocking on acquiring the row-level read lock. This is because the read lock can only block if someone else holds the write lock, but both the write and read lock are only acquired under the row lock. I ran a quick test of this approach over a region (this directly interacts with HRegion, so no client effects): - 30 threads - 5000 increments per thread - 30 columns per increment - Each increment uniformly distributed over 500,000 rows - 5 trials Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms Today: 13950 ms The locking approach: 10877 ms So it looks like an improvement, at least wrt increment. As mentioned, I need to measure the overhead of acquiring the read lock for puts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5416) Improve performance of scans with some kind of filters.
[ https://issues.apache.org/jira/browse/HBASE-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540105#comment-13540105 ] Hadoop QA commented on HBASE-5416: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562493/5416-v13.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 9 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2 warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces lines longer than 100 {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster org.apache.hadoop.hbase.replication.TestReplication {color:red}-1 core zombie tests{color}. There are 4 zombie test(s): Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3718//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3718//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3718//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3718//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3718//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3718//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3718//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3718//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3718//console This message is automatically generated. Improve performance of scans with some kind of filters. --- Key: HBASE-5416 URL: https://issues.apache.org/jira/browse/HBASE-5416 Project: HBase Issue Type: Improvement Components: Filters, Performance, regionserver Affects Versions: 0.90.4 Reporter: Max Lapan Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: 5416-Filtered_scans_v6.patch, 5416-v13.patch, 5416-v5.txt, 5416-v6.txt, Filtered_scans.patch, Filtered_scans_v2.patch, Filtered_scans_v3.patch, Filtered_scans_v4.patch, Filtered_scans_v5.1.patch, Filtered_scans_v5.patch, Filtered_scans_v7.patch, HBASE-5416-v10.patch, HBASE-5416-v11.patch, HBASE-5416-v12.patch, HBASE-5416-v12.patch, HBASE-5416-v7-rebased.patch, HBASE-5416-v8.patch, HBASE-5416-v9.patch When the scan is performed, whole row is loaded into result list, after that filter (if exists) is applied to detect that row is needed. But when scan is performed on several CFs and filter checks only data from the subset of these CFs, data from CFs, not checked by a filter is not needed on a filter stage. Only when we decided to include current row. And in such case we can significantly reduce amount of IO performed by a scan, by loading only values, actually checked by a filter. For example, we have two CFs: flags and snap. Flags is quite small (bunch of megabytes) and is used to filter large entries from snap. Snap is very large (10s of GB) and it is quite costly to scan it. If we needed only rows with some flag specified, we use SingleColumnValueFilter to limit result to only small subset of region. But current implementation is loading both CFs to perform scan, when only small subset is needed. Attached patch adds one routine to Filter interface to allow filter to specify which CF is needed to it's operation. In HRegion, we separate all scanners into two groups: needed for filter and the rest (joined). When new row is considered, only needed data is loaded, filter applied, and only if
[jira] [Created] (HBASE-7444) [89-fb] Update the default user name in MiniHBaseCluster
Liyin Tang created HBASE-7444: - Summary: [89-fb] Update the default user name in MiniHBaseCluster Key: HBASE-7444 URL: https://issues.apache.org/jira/browse/HBASE-7444 Project: HBase Issue Type: Test Reporter: Liyin Tang Priority: Minor Currently we are using $username.hrs.$index as default user name in MiniHBaseCluster, which actually is not a legal user name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-5945: --- Attachment: 5945-in-progress.patch [~stack], hope it is okay with you that I am attaching a patch here... (I was tempted to do a patch for this issue..) The main change in this patch is that it removes the RpcRequestBody from RPC.proto, and instead serializes the rpc method argument directly to the underlying output. There is a lot of change associated with this removal in the patch. I didn't change the serialization of the rpc header fields yet (as was done by [~tlipcon] in his earlier patch). The reason being that I don't think the header part is a concern since they will be a few 10s of bytes and mostly will be noise. The ser/de of the rpc request-body/response-body is where I tried to improve on. The codebase compiles with this patch now but there is work still left (and maybe I missed some buffer copies as well; need to dig some more). Reduce buffer copies in IPC server response path Key: HBASE-5945 URL: https://issues.apache.org/jira/browse/HBASE-5945 Project: HBase Issue Type: Improvement Components: IPC/RPC Affects Versions: 0.96.0 Reporter: Todd Lipcon Assignee: stack Priority: Blocker Fix For: 0.96.0 Attachments: 5945-in-progress.patch, buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt The new PB code is sloppy with buffers and makes several needless copies. This increases GC time a lot. A few simple changes can cut this back down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5705) Introduce Protocol Buffer RPC engine
[ https://issues.apache.org/jira/browse/HBASE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540138#comment-13540138 ] stack commented on HBASE-5705: -- It looks like this patch made it so that when we set up the proxy, we no longer ask the server what its version of the protocol is. Instead we return the client's version of the protocol. See below. + static long getProtocolVersion(Class? extends VersionedProtocol protocol) + throws NoSuchFieldException, IllegalAccessException { +Field versionField = protocol.getField(VERSION); +versionField.setAccessible(true); +return versionField.getLong(protocol); + } Was expediency the reason the querying of server version was undone? Something to be addressed later when VersionedProtocol was fixed up? Thanks. Introduce Protocol Buffer RPC engine Key: HBASE-5705 URL: https://issues.apache.org/jira/browse/HBASE-5705 Project: HBase Issue Type: Sub-task Components: IPC/RPC, master, migration, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 5705-1.patch, 5705-2.1.patch, 5705-2.2.patch, 5705-2.3.patch Introduce Protocol Buffer RPC engine in the RPC core. Protocols that are PB aware can be made to go through this RPC engine. The approach, in my current thinking, would be similar to HADOOP-7773. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7188) Move classes into hbase-client
[ https://issues.apache.org/jira/browse/HBASE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540139#comment-13540139 ] stack commented on HBASE-7188: -- Thanks [~eclark] Looks like something we need to reinstitute. Will do over in hbase-6521. Move classes into hbase-client -- Key: HBASE-7188 URL: https://issues.apache.org/jira/browse/HBASE-7188 Project: HBase Issue Type: Sub-task Components: Client, IPC/RPC Affects Versions: 0.96.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.96.0 Attachments: HBASE-7188-0.patch, HBASE-7188-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7188) Move classes into hbase-client
[ https://issues.apache.org/jira/browse/HBASE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540145#comment-13540145 ] stack commented on HBASE-7188: -- It didn't apply clean for me. patching file hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java Reversed (or previously applied) patch detected! Assume -R? [n] Am I seeing that properly, that there are packages in hbase-common now for regionserver and security? Doesn't seem right. Should hbase-default.xml be in hbase-common? Or that version stuff Move classes into hbase-client -- Key: HBASE-7188 URL: https://issues.apache.org/jira/browse/HBASE-7188 Project: HBase Issue Type: Sub-task Components: Client, IPC/RPC Affects Versions: 0.96.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.96.0 Attachments: HBASE-7188-0.patch, HBASE-7188-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically)
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540146#comment-13540146 ] Himanshu Vashishtha commented on HBASE-7055: I think it would be nice to have some benchmarks for this feature against the existing Compaction algo? port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically) - Key: HBASE-7055 URL: https://issues.apache.org/jira/browse/HBASE-7055 Project: HBase Issue Type: Task Components: Compaction Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, HBASE-6371-v3-refactor-only-squashed.patch, HBASE-6371-v4-refactor-only-squashed.patch, HBASE-6371-v5-refactor-only-squashed.patch, HBASE-7055-v0.patch, HBASE-7055-v1.patch, HBASE-7055-v2.patch, HBASE-7055-v3.patch, HBASE-7055-v4.patch There's divergence in the code :( See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7442) HBase remote CopyTable not working when security enabled
[ https://issues.apache.org/jira/browse/HBASE-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540152#comment-13540152 ] Andrew Purtell commented on HBASE-7442: --- Thanks for identifying the problem. We should fix this (incorrect) assumption made in the client code. I don't have a strong opinion but I like option #1 better: for the sake of simplicity a connection to a regionserver is handled (and cached) the same way whether it is the local cluster or remote. HBase remote CopyTable not working when security enabled Key: HBASE-7442 URL: https://issues.apache.org/jira/browse/HBASE-7442 Project: HBase Issue Type: Bug Components: IPC/RPC, mapreduce, security Affects Versions: 0.92.1 Reporter: James Kinley Fix For: 0.96.0, 0.94.5 Attachments: attempt_201212271546_0001_m_00_0.log When security is enabled, HBase CopyTable fails with Kerberos exception: {code} FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] {code} This is only when copying to remote HBase cluster (using either MRv1 or YARN), local copy works fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-7442) HBase remote CopyTable not working when security enabled
[ https://issues.apache.org/jira/browse/HBASE-7442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540152#comment-13540152 ] Andrew Purtell edited comment on HBASE-7442 at 12/27/12 9:08 PM: - Thanks for identifying the problem. We should fix this (incorrect) assumption made in the client code. I don't have a strong opinion but I like option #1 better: for the sake of simplicity a connection is handled (and cached) the same way whether it is to a host in the local cluster or a remote one. was (Author: apurtell): Thanks for identifying the problem. We should fix this (incorrect) assumption made in the client code. I don't have a strong opinion but I like option #1 better: for the sake of simplicity a connection to a regionserver is handled (and cached) the same way whether it is the local cluster or remote. HBase remote CopyTable not working when security enabled Key: HBASE-7442 URL: https://issues.apache.org/jira/browse/HBASE-7442 Project: HBase Issue Type: Bug Components: IPC/RPC, mapreduce, security Affects Versions: 0.92.1 Reporter: James Kinley Fix For: 0.96.0, 0.94.5 Attachments: attempt_201212271546_0001_m_00_0.log When security is enabled, HBase CopyTable fails with Kerberos exception: {code} FATAL org.apache.hadoop.ipc.SecureClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] {code} This is only when copying to remote HBase cluster (using either MRv1 or YARN), local copy works fine. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7443) More findbugs fixes
[ https://issues.apache.org/jira/browse/HBASE-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540161#comment-13540161 ] stack commented on HBASE-7443: -- +1 on fix. Hoist note on new annotation up into the release notes I'd say on commit. Yes, the initialize while class is being constructed is a tricky one findbugs finding real issues. More findbugs fixes --- Key: HBASE-7443 URL: https://issues.apache.org/jira/browse/HBASE-7443 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 0.96.0 Attachments: 7443.patch It - adds the dependency to findbugs for the annotations. It's a compile only dependency. License is LGPL. - suppresses a few not critical warnings - fixes a few others Locally, I'm now at 144. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically)
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540176#comment-13540176 ] Jimmy Xiang commented on HBASE-7055: I like the idea of this feature. I think it is useful. However, I am not sure how to use it, for example, how many tiers a region should have, how to separate each tier, and why? This patch introduces several configuration parameters to separate tiers. Do we have a guideline on how to use these parameters? What kind tier separation will make the compaction scheduling efficient? If the tiers are not set properly, could it make things even worse instead? By the tier term, the existing compaction selection has just one tier. Can we automatically skip the most recent storefile till we get a valid selection? By this way, we can have dynamical tiers which guarantees some valid selection? port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically) - Key: HBASE-7055 URL: https://issues.apache.org/jira/browse/HBASE-7055 Project: HBase Issue Type: Task Components: Compaction Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, HBASE-6371-v3-refactor-only-squashed.patch, HBASE-6371-v4-refactor-only-squashed.patch, HBASE-6371-v5-refactor-only-squashed.patch, HBASE-7055-v0.patch, HBASE-7055-v1.patch, HBASE-7055-v2.patch, HBASE-7055-v3.patch, HBASE-7055-v4.patch There's divergence in the code :( See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6907) KeyValue equals and compareTo methods should match
[ https://issues.apache.org/jira/browse/HBASE-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540212#comment-13540212 ] Ted Yu commented on HBASE-6907: --- Over in https://reviews.apache.org/r/7591 (for HBASE-4676), memstoreTS is named MvccVersion KeyValue equals and compareTo methods should match -- Key: HBASE-6907 URL: https://issues.apache.org/jira/browse/HBASE-6907 Project: HBase Issue Type: Bug Components: util Reporter: Matt Corgan Priority: Critical Fix For: 0.96.0 KeyValue.KVComparator includes the memstoreTS when comparing, however the KeyValue.equals() method ignores the memstoreTS. The Comparator interface has always specified that comparator return 0 when equals would return true and vice versa. Obeying that rule has been sort of optional in the past, but Java 7 introduces a new default collection sorting algorithm called Tim Sort which relies on that behavior. http://bugs.sun.com/view_bug.do?bug_id=6804124 Possible problem spots: * there's a Collections.sort(KeyValues) in RedundantKVGenerator.generateTestKeyValues(..) * TestColumnSeeking compares two collections of KeyValues using the containsAll method. It is intentionally ignoring memstoreTS, so will need an alternative method for comparing the two collections. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5776) HTableMultiplexer
[ https://issues.apache.org/jira/browse/HBASE-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540222#comment-13540222 ] Hudson commented on HBASE-5776: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #315 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/315/]) HBASE-5776 HTableMultiplexer (Liyin, binlijin and Ted Yu) (Revision 1426221) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java HTableMultiplexer -- Key: HBASE-5776 URL: https://issues.apache.org/jira/browse/HBASE-5776 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: binlijin Fix For: 0.96.0 Attachments: 5776-trunk-V3.patch, 5776-trunk-V4.patch, 5776-trunk-V5.patch, 5776-trunk-V6.patch, 5776-trunk-V7.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.1.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.2.patch, ASF.LICENSE.NOT.GRANTED--D2775.3.patch, ASF.LICENSE.NOT.GRANTED--D2775.4.patch, ASF.LICENSE.NOT.GRANTED--D2775.5.patch, HBASE-5776-trunk.patch, HBASE-5776-trunk-V2.patch There is a known issue in HBase client that single slow/dead region server could slow down the multiput operations across all the region servers. So the HBase client will be as slow as the slowest region server in the cluster. To solve this problem, HTableMultiplexer will separate the multiput submitting threads with the flush threads, which means the multiput operation will be a nonblocking operation. The submitting thread will shard all the puts into different queues based on its destination region server and return immediately. The flush threads will flush these puts from each queue to its destination region server. Currently the HTableMultiplexer only supports the put operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7188) Move classes into hbase-client
[ https://issues.apache.org/jira/browse/HBASE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-7188: - Attachment: HBASE-7188-2.patch Rebased patch. Regarding having region server stuff in common: * Yeah mostly exceptions are in there since they are shared. The client needs to be able to de-serialize them if they are thrown over the wire. * BloomType should move as well but this is such a huge patch that I wanted to keep the renamig and moving to a minimum as keeping it rebased to trunk is a good deal of work. Security: This can either be in common or client. It's shared between the server and the client so I thought common but it can go either place. Move classes into hbase-client -- Key: HBASE-7188 URL: https://issues.apache.org/jira/browse/HBASE-7188 Project: HBase Issue Type: Sub-task Components: Client, IPC/RPC Affects Versions: 0.96.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.96.0 Attachments: HBASE-7188-0.patch, HBASE-7188-1.patch, HBASE-7188-2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7188) Move classes into hbase-client
[ https://issues.apache.org/jira/browse/HBASE-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540230#comment-13540230 ] stack commented on HBASE-7188: -- Can the shared exceptions be anonymized? i.e. not particular to the regionserver and therefore not have to be in the regionserver subpackage? Ok on bloomtype. A 'generic' security in common is probably fine. Move classes into hbase-client -- Key: HBASE-7188 URL: https://issues.apache.org/jira/browse/HBASE-7188 Project: HBase Issue Type: Sub-task Components: Client, IPC/RPC Affects Versions: 0.96.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.96.0 Attachments: HBASE-7188-0.patch, HBASE-7188-1.patch, HBASE-7188-2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7410) [snapshots] add snapshot/clone/restore/export docs to ref guide
[ https://issues.apache.org/jira/browse/HBASE-7410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7410: --- Attachment: HBASE-7410-v0.patch Added some notes, that should allow to start using snapshot. [snapshots] add snapshot/clone/restore/export docs to ref guide --- Key: HBASE-7410 URL: https://issues.apache.org/jira/browse/HBASE-7410 Project: HBase Issue Type: Sub-task Components: documentation, snapshots Affects Versions: hbase-6055 Reporter: Jonathan Hsieh Priority: Blocker Fix For: hbase-6055, 0.96.0 Attachments: HBASE-7410-v0.patch This will include additions to the ref guide about the different operations provided and how to use them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7294) Check for snapshot file cleaners on start
[ https://issues.apache.org/jira/browse/HBASE-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540242#comment-13540242 ] Jonathan Hsieh commented on HBASE-7294: --- looks good. +1. Running through tests and will commit if they pass. Check for snapshot file cleaners on start - Key: HBASE-7294 URL: https://issues.apache.org/jira/browse/HBASE-7294 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Affects Versions: hbase-6055 Reporter: Jesse Yates Assignee: Matteo Bertozzi Fix For: hbase-6055, 0.96.0 Attachments: HBASE-7294-v1.patch, HBASE-7294-v2.patch, HBASE-7294-v3.patch, HBASE-7294-v4.patch, HBASE-7294-v5.patch, HBASE-7294-v6.patch Snapshots currently use the SnaphotHfileCleaner and SnapshotHLogCleaner to ensure that any hfiles or hlogs (respectively) that are currently part of a snapshot are not removed from their respective archive directories (.archive and .oldlogs). From Matteo Bertozzi: {quote} currently the snapshot cleaner is not in hbase-default.xml and there's no warning/exception on snapshot/restore operation, if not enabled. even if we add the cleaner to the hbase-default.xml how do we ensure that the user doesn't remove it? Do we want to hardcode the cleaner at master startup? Do we want to add a check in snapshot/restore that throws an exception if the cleaner is not enabled? {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540257#comment-13540257 ] Hadoop QA commented on HBASE-7403: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12562362/7403-trunkv5.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 10 warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces lines longer than 100 {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.TestGlobalMemStoreSize org.apache.hadoop.hbase.mapreduce.TestRowCounter org.apache.hadoop.hbase.replication.TestMasterReplication org.apache.hadoop.hbase.TestFullLogReconstruction org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles org.apache.hadoop.hbase.TestClusterBootOrder org.apache.hadoop.hbase.mapreduce.TestMultithreadedTableMapper org.apache.hadoop.hbase.replication.TestMultiSlaveReplication org.apache.hadoop.hbase.coprocessor.TestRowProcessorEndpoint org.apache.hadoop.hbase.client.TestHCM org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.ipc.TestDelayedRpc org.apache.hadoop.hbase.mapreduce.TestImportExport {color:red}-1 core zombie tests{color}. There are 5 zombie test(s): at org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testExcludeAllFromMinorCompaction(TestHFileOutputFormat.java:693) at org.apache.hadoop.hbase.util.TestHBaseFsck.testMissingFirstRegion(TestHBaseFsck.java:1281) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3722//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3722//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3722//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3722//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3722//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3722//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3722//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3722//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3722//console This message is automatically generated. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv5.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to care event like Server Dead, Balance, Split, Disabing/Enabing table, no need to care whether you send a wrong merge request, it alread have done for you 5.Only little offline time for two merging regions We need merge in the following cases: 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey
[jira] [Updated] (HBASE-7384) Introducing waitForCondition function into test cases
[ https://issues.apache.org/jira/browse/HBASE-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-7384: - Attachment: hbase-7384.patch This patch is a baby step towards on flaky tests. Most flaky tests I found especially for cross platform tests are due to not long enough hard coded time out values. It would be much more pleasant if we can easily identify premature time out situations and adjust max time out values dynamically by a system property according to different environment needs. I migrated two test cases to use the new waitFor functions as an example. The new waitFor function will fully benefit us after we migrate all old test cases and use the new waitFor pattern for future test cases. Thanks, -Jeffrey Introducing waitForCondition function into test cases - Key: HBASE-7384 URL: https://issues.apache.org/jira/browse/HBASE-7384 Project: HBase Issue Type: Test Components: test Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Attachments: hbase-7384.patch, Waiter.java Recently I'm working on flaky test cases and found we have many places using while loop and sleep to wait for a condition to be true. There are several issues in existing ways: 1) Many similar code doing the same thing 2) When time out happens, different errors are reported without explicitly indicating a time out situation 3) When we want to increase the max timeout value to verify if a test case fails due to a not-enough time out value, we have to recompile redeploy code I propose to create a waitForCondition function as a test utility function like the following: {code} public interface WaitCheck { public boolean Check() ; } public boolean waitForCondition(int timeOutInMilliSeconds, int checkIntervalInMilliSeconds, WaitCheck s) throws InterruptedException { int multiplier = 1; String multiplierProp = System.getProperty(extremeWaitMultiplier); if(multiplierProp != null) { multiplier = Integer.parseInt(multiplierProp); if(multiplier 1) { LOG.warn(String.format(Invalid extremeWaitMultiplier property value:%s. is ignored., multiplierProp)); multiplier = 1; } } int timeElapsed = 0; while(timeElapsed timeOutInMilliSeconds * multiplier) { if(s.Check()) { return true; } Thread.sleep(checkIntervalInMilliSeconds); timeElapsed += checkIntervalInMilliSeconds; } assertTrue(WaitForCondition failed due to time out( + timeOutInMilliSeconds + milliseconds expired), false); return false; } {code} By doing the above way, there are several advantages: 1) Clearly report time out error when such situation happens 2) Use System property extremeWaitMultiplier to increase max time out dynamically for a quick verification 3) Standardize current wait situations Pleas let me know what your thoughts on this. Thanks, -Jeffrey -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7384) Introducing waitForCondition function into test cases
[ https://issues.apache.org/jira/browse/HBASE-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-7384: - Fix Version/s: 0.96.0 Labels: test (was: ) Status: Patch Available (was: Open) Introducing waitForCondition function into test cases - Key: HBASE-7384 URL: https://issues.apache.org/jira/browse/HBASE-7384 Project: HBase Issue Type: Test Components: test Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Labels: test Fix For: 0.96.0 Attachments: hbase-7384.patch, Waiter.java Recently I'm working on flaky test cases and found we have many places using while loop and sleep to wait for a condition to be true. There are several issues in existing ways: 1) Many similar code doing the same thing 2) When time out happens, different errors are reported without explicitly indicating a time out situation 3) When we want to increase the max timeout value to verify if a test case fails due to a not-enough time out value, we have to recompile redeploy code I propose to create a waitForCondition function as a test utility function like the following: {code} public interface WaitCheck { public boolean Check() ; } public boolean waitForCondition(int timeOutInMilliSeconds, int checkIntervalInMilliSeconds, WaitCheck s) throws InterruptedException { int multiplier = 1; String multiplierProp = System.getProperty(extremeWaitMultiplier); if(multiplierProp != null) { multiplier = Integer.parseInt(multiplierProp); if(multiplier 1) { LOG.warn(String.format(Invalid extremeWaitMultiplier property value:%s. is ignored., multiplierProp)); multiplier = 1; } } int timeElapsed = 0; while(timeElapsed timeOutInMilliSeconds * multiplier) { if(s.Check()) { return true; } Thread.sleep(checkIntervalInMilliSeconds); timeElapsed += checkIntervalInMilliSeconds; } assertTrue(WaitForCondition failed due to time out( + timeOutInMilliSeconds + milliseconds expired), false); return false; } {code} By doing the above way, there are several advantages: 1) Clearly report time out error when such situation happens 2) Use System property extremeWaitMultiplier to increase max time out dynamically for a quick verification 3) Standardize current wait situations Pleas let me know what your thoughts on this. Thanks, -Jeffrey -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7384) Introducing waitForCondition function into test cases
[ https://issues.apache.org/jira/browse/HBASE-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-7384: - Attachment: hbase-7384_1.0.patch Introducing waitForCondition function into test cases - Key: HBASE-7384 URL: https://issues.apache.org/jira/browse/HBASE-7384 Project: HBase Issue Type: Test Components: test Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Labels: test Fix For: 0.96.0 Attachments: hbase-7384_1.0.patch, hbase-7384.patch, Waiter.java Recently I'm working on flaky test cases and found we have many places using while loop and sleep to wait for a condition to be true. There are several issues in existing ways: 1) Many similar code doing the same thing 2) When time out happens, different errors are reported without explicitly indicating a time out situation 3) When we want to increase the max timeout value to verify if a test case fails due to a not-enough time out value, we have to recompile redeploy code I propose to create a waitForCondition function as a test utility function like the following: {code} public interface WaitCheck { public boolean Check() ; } public boolean waitForCondition(int timeOutInMilliSeconds, int checkIntervalInMilliSeconds, WaitCheck s) throws InterruptedException { int multiplier = 1; String multiplierProp = System.getProperty(extremeWaitMultiplier); if(multiplierProp != null) { multiplier = Integer.parseInt(multiplierProp); if(multiplier 1) { LOG.warn(String.format(Invalid extremeWaitMultiplier property value:%s. is ignored., multiplierProp)); multiplier = 1; } } int timeElapsed = 0; while(timeElapsed timeOutInMilliSeconds * multiplier) { if(s.Check()) { return true; } Thread.sleep(checkIntervalInMilliSeconds); timeElapsed += checkIntervalInMilliSeconds; } assertTrue(WaitForCondition failed due to time out( + timeOutInMilliSeconds + milliseconds expired), false); return false; } {code} By doing the above way, there are several advantages: 1) Clearly report time out error when such situation happens 2) Use System property extremeWaitMultiplier to increase max time out dynamically for a quick verification 3) Standardize current wait situations Pleas let me know what your thoughts on this. Thanks, -Jeffrey -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7445) Hbase cluster is unavailable while the regionserver that Meta table deployed crashed
Zhong Deyin created HBASE-7445: -- Summary: Hbase cluster is unavailable while the regionserver that Meta table deployed crashed Key: HBASE-7445 URL: https://issues.apache.org/jira/browse/HBASE-7445 Project: HBase Issue Type: Bug Components: Region Assignment, regionserver Affects Versions: 0.94.1 Environment: Hadoop 0.20.2-cdh3u3 Hbase 0.94.1 Reporter: Zhong Deyin Fix For: 0.94.1 while the regionserver that META table deployed crashed, the .META. table can't migrate to other available regionservers. Then the region spliting, can't find META table, cause the whole cluster is unavailable. Code path: org.apache.hadoop.hbase.master.handler.ServerShutdownHandler {code} // Carrying meta if (isCarryingMeta()) { LOG.info(Server + serverName + was carrying META. Trying to assign.); this.services.getAssignmentManager(). regionOffline(HRegionInfo.FIRST_META_REGIONINFO); this.services.getAssignmentManager().assignMeta(); } {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7445) Hbase cluster is unavailable while the regionserver that Meta table deployed crashed
[ https://issues.apache.org/jira/browse/HBASE-7445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Deyin updated HBASE-7445: --- Fix Version/s: (was: 0.94.1) Hbase cluster is unavailable while the regionserver that Meta table deployed crashed Key: HBASE-7445 URL: https://issues.apache.org/jira/browse/HBASE-7445 Project: HBase Issue Type: Bug Components: Region Assignment, regionserver Affects Versions: 0.94.1 Environment: Hadoop 0.20.2-cdh3u3 Hbase 0.94.1 Reporter: Zhong Deyin Labels: patch Original Estimate: 336h Remaining Estimate: 336h while the regionserver that META table deployed crashed, the .META. table can't migrate to other available regionservers. Then the region spliting, can't find META table, cause the whole cluster is unavailable. Code path: org.apache.hadoop.hbase.master.handler.ServerShutdownHandler {code} // Carrying meta if (isCarryingMeta()) { LOG.info(Server + serverName + was carrying META. Trying to assign.); this.services.getAssignmentManager(). regionOffline(HRegionInfo.FIRST_META_REGIONINFO); this.services.getAssignmentManager().assignMeta(); } {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7384) Introducing waitForCondition function into test cases
[ https://issues.apache.org/jira/browse/HBASE-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540276#comment-13540276 ] Enis Soztutar commented on HBASE-7384: -- This tool is useful, but as Nick pointed out, we should really be using CountDownLatch'es for test synchronization. The problem, however, is that most of the time, the latch release should happen from the main code, and there is no generic injection mechanism yet to do this from the tests cleanly. For example the patch in HBASE-5494 introduces a class called InjectionHandler for doing this, but not sure if it is the best interface. Above is out of the scope for this isssue, but I think we should keep that in mind. Introducing waitForCondition function into test cases - Key: HBASE-7384 URL: https://issues.apache.org/jira/browse/HBASE-7384 Project: HBase Issue Type: Test Components: test Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Labels: test Fix For: 0.96.0 Attachments: hbase-7384_1.0.patch, hbase-7384.patch, Waiter.java Recently I'm working on flaky test cases and found we have many places using while loop and sleep to wait for a condition to be true. There are several issues in existing ways: 1) Many similar code doing the same thing 2) When time out happens, different errors are reported without explicitly indicating a time out situation 3) When we want to increase the max timeout value to verify if a test case fails due to a not-enough time out value, we have to recompile redeploy code I propose to create a waitForCondition function as a test utility function like the following: {code} public interface WaitCheck { public boolean Check() ; } public boolean waitForCondition(int timeOutInMilliSeconds, int checkIntervalInMilliSeconds, WaitCheck s) throws InterruptedException { int multiplier = 1; String multiplierProp = System.getProperty(extremeWaitMultiplier); if(multiplierProp != null) { multiplier = Integer.parseInt(multiplierProp); if(multiplier 1) { LOG.warn(String.format(Invalid extremeWaitMultiplier property value:%s. is ignored., multiplierProp)); multiplier = 1; } } int timeElapsed = 0; while(timeElapsed timeOutInMilliSeconds * multiplier) { if(s.Check()) { return true; } Thread.sleep(checkIntervalInMilliSeconds); timeElapsed += checkIntervalInMilliSeconds; } assertTrue(WaitForCondition failed due to time out( + timeOutInMilliSeconds + milliseconds expired), false); return false; } {code} By doing the above way, there are several advantages: 1) Clearly report time out error when such situation happens 2) Use System property extremeWaitMultiplier to increase max time out dynamically for a quick verification 3) Standardize current wait situations Pleas let me know what your thoughts on this. Thanks, -Jeffrey -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7445) Hbase cluster is unavailable while the regionserver that Meta table deployed crashed
[ https://issues.apache.org/jira/browse/HBASE-7445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Deyin resolved HBASE-7445. Resolution: Fixed modify class org.apache.hadoop.hbase.master.handler.ServerShutdownHandler, change the process method, replace code this.services.getAssignmentManager().assignMeta() to assignMetaWithRetries(),then meta will try 10 times while the regionservers crashed. {code} // Carrying meta if (isCarryingMeta()) { LOG.info(Server + serverName + was carrying META. Trying to assign.); this.services.getAssignmentManager(). regionOffline(HRegionInfo.FIRST_META_REGIONINFO); //this.services.getAssignmentManager().assignMeta(); assignMetaWithRetries(); } {code} Add method assignMetaWithRetries, code of assignMetaWithRetries method as follows: {code} private void assignMetaWithRetries() throws IOException{ int iTimes = this.server.getConfiguration().getInt( hbase.catalog.verification.retries, 10); long waitTime = this.server.getConfiguration().getLong( hbase.catalog.verification.timeout, 1000); int iFlag = 0; LOG.info(TEST START); while (true) { try { // verifyAndAssignRoot(); this.services.getAssignmentManager().assignMeta(); break; } catch (Exception e) { if (iFlag = iTimes) { this.server.abort( test ming assginMeta failed after + iTimes + times retries, aborting, e); throw new IOException(Aborting, e); } try { Thread.sleep(waitTime); } catch (InterruptedException e1) { LOG.warn(Interrupted when is the thread sleep, e1); Thread.currentThread().interrupt(); throw new IOException(Interrupted, e1); } iFlag++; } } LOG.info(TEST END HBASE); } {code} Hbase cluster is unavailable while the regionserver that Meta table deployed crashed Key: HBASE-7445 URL: https://issues.apache.org/jira/browse/HBASE-7445 Project: HBase Issue Type: Bug Components: Region Assignment, regionserver Affects Versions: 0.94.1 Environment: Hadoop 0.20.2-cdh3u3 Hbase 0.94.1 Reporter: Zhong Deyin Labels: patch Original Estimate: 336h Remaining Estimate: 336h while the regionserver that META table deployed crashed, the .META. table can't migrate to other available regionservers. Then the region spliting, can't find META table, cause the whole cluster is unavailable. Code path: org.apache.hadoop.hbase.master.handler.ServerShutdownHandler {code} // Carrying meta if (isCarryingMeta()) { LOG.info(Server + serverName + was carrying META. Trying to assign.); this.services.getAssignmentManager(). regionOffline(HRegionInfo.FIRST_META_REGIONINFO); this.services.getAssignmentManager().assignMeta(); } {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-7445) Hbase cluster is unavailable while the regionserver that Meta table deployed crashed
[ https://issues.apache.org/jira/browse/HBASE-7445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Deyin reopened HBASE-7445: Hbase cluster is unavailable while the regionserver that Meta table deployed crashed Key: HBASE-7445 URL: https://issues.apache.org/jira/browse/HBASE-7445 Project: HBase Issue Type: Bug Components: Region Assignment, regionserver Affects Versions: 0.94.1 Environment: Hadoop 0.20.2-cdh3u3 Hbase 0.94.1 Reporter: Zhong Deyin Labels: patch Original Estimate: 336h Remaining Estimate: 336h while the regionserver that META table deployed crashed, the .META. table can't migrate to other available regionservers. Then the region spliting, can't find META table, cause the whole cluster is unavailable. Code path: org.apache.hadoop.hbase.master.handler.ServerShutdownHandler {code} // Carrying meta if (isCarryingMeta()) { LOG.info(Server + serverName + was carrying META. Trying to assign.); this.services.getAssignmentManager(). regionOffline(HRegionInfo.FIRST_META_REGIONINFO); this.services.getAssignmentManager().assignMeta(); } {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5705) Introduce Protocol Buffer RPC engine
[ https://issues.apache.org/jira/browse/HBASE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540306#comment-13540306 ] Devaraj Das commented on HBASE-5705: [~stack], yes. The idea was to have the server decide how to best handle the rpc method call based on the client's protocol version. Though the server currently ignores the version, this is something that can be used to implement versioning. Please have a look at ProtobufRpcServerEngine.java around the area with the comment TODO: use the clientVersion... Other than that, HBASE-6521 should address versioning in a more complete fashion (and I guess we should unify all the versioning stories there). Introduce Protocol Buffer RPC engine Key: HBASE-5705 URL: https://issues.apache.org/jira/browse/HBASE-5705 Project: HBase Issue Type: Sub-task Components: IPC/RPC, master, migration, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 5705-1.patch, 5705-2.1.patch, 5705-2.2.patch, 5705-2.3.patch Introduce Protocol Buffer RPC engine in the RPC core. Protocols that are PB aware can be made to go through this RPC engine. The approach, in my current thinking, would be similar to HADOOP-7773. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540307#comment-13540307 ] Ted Yu commented on HBASE-3776: --- I already gave my +1. One more +1 is needed. Add Bloom Filter Support to HFileOutputFormat - Key: HBASE-3776 URL: https://issues.apache.org/jira/browse/HBASE-3776 Project: HBase Issue Type: Sub-task Components: mapreduce Reporter: Nicolas Spiegelberg Assignee: Anoop Sam John Priority: Critical Labels: hbase Fix For: 0.96.0 Attachments: HBASE-3776.patch, HBASE-3776_v2.patch Add Bloom Filter support for bulk imports. Lacking a bloom filter, even on a single imported file, can cause perf degradation. Since we now set our compression type based on the HBase CF configuration, it would be good to follow this path for the bloom filter addition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7446) Make the reference guide utf8 rather than 8859
[ https://issues.apache.org/jira/browse/HBASE-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7446: - Attachment: utf8.txt Here is a patch that seems to do the right thing. Make the reference guide utf8 rather than 8859 -- Key: HBASE-7446 URL: https://issues.apache.org/jira/browse/HBASE-7446 Project: HBase Issue Type: Bug Reporter: stack Attachments: utf8.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540303#comment-13540303 ] Anoop Sam John commented on HBASE-3776: --- @Ted, @stack - Is it good to go? Add Bloom Filter Support to HFileOutputFormat - Key: HBASE-3776 URL: https://issues.apache.org/jira/browse/HBASE-3776 Project: HBase Issue Type: Sub-task Components: mapreduce Reporter: Nicolas Spiegelberg Assignee: Anoop Sam John Priority: Critical Labels: hbase Fix For: 0.96.0 Attachments: HBASE-3776.patch, HBASE-3776_v2.patch Add Bloom Filter support for bulk imports. Lacking a bloom filter, even on a single imported file, can cause perf degradation. Since we now set our compression type based on the HBase CF configuration, it would be good to follow this path for the bloom filter addition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7446) Make the reference guide utf8 rather than 8859
[ https://issues.apache.org/jira/browse/HBASE-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-7446. -- Resolution: Fixed Fix Version/s: 0.96.0 Assignee: stack Committed to trunk. Make the reference guide utf8 rather than 8859 -- Key: HBASE-7446 URL: https://issues.apache.org/jira/browse/HBASE-7446 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.96.0 Attachments: utf8.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7446) Make the reference guide utf8 rather than 8859
stack created HBASE-7446: Summary: Make the reference guide utf8 rather than 8859 Key: HBASE-7446 URL: https://issues.apache.org/jira/browse/HBASE-7446 Project: HBase Issue Type: Bug Reporter: stack Attachments: utf8.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6132) ColumnCountGetFilter PageFilter not working with FilterList
[ https://issues.apache.org/jira/browse/HBASE-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540304#comment-13540304 ] Anoop Sam John commented on HBASE-6132: --- Devs I feel the attached test case reveals the problem with FilterList. If all agree I can make the change and submit a patch. [~saint@gmail.com] Pls check the test case once ColumnCountGetFilter PageFilter not working with FilterList - Key: HBASE-6132 URL: https://issues.apache.org/jira/browse/HBASE-6132 Project: HBase Issue Type: Bug Components: Filters Affects Versions: 0.92.0, 0.92.1, 0.94.0 Environment: Cent OS 5.5 distributed hbase cluster. Hadoop 1.0.0, zookeeper 3.4.3 Reporter: Benjamin Kim Assignee: Anoop Sam John Attachments: TestColumnCountGetFilter.java Thanks to Anoop and Ramkrishna, here's what we found with FilterList If I use FilterList to include ColumnCountGetFilter among other filters, the returning Result has no keyvalues. This problem seems to occur when specified column count is less then actual number of existing columns. Also same problem arises with PageFilter Following is the code of the problem: {code} Configuration conf = HBaseConfiguration.create(); HTable table = new HTable(conf, test); Get get = new Get(Bytes.toBytes(test1)); FilterList filterList = new FilterList(); filterList.addFilter(new ColumnCountGetFilter(100)); get.setFilter(filterList); Result r = table.get(get); System.out.println(r.size()); // prints zero {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7447) Occasional failures of TestSplitLogWorker#testAcquireTaskAtStartup
Andrew Purtell created HBASE-7447: - Summary: Occasional failures of TestSplitLogWorker#testAcquireTaskAtStartup Key: HBASE-7447 URL: https://issues.apache.org/jira/browse/HBASE-7447 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.4 Reporter: Andrew Purtell I see occasional failures (1 out of ~20) on branch 0.94 of TestSplitLogWorker#testAcquireTaskAtStartup {noformat} Running org.apache.hadoop.hbase.regionserver.TestSplitLogWorker Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.737 sec FAILURE! Failed tests: testAcquireTaskAtStartup(org.apache.hadoop.hbase.regionserver.TestSplitLogWorker): ctr=0, oldval=0, newval=1 {noformat} Bisecting commits on 0.94 branch, using (up to) 50 repetitions to include or exclude, I get: {noformat} 8cc5a5f481aeaf6debfa95110c4392d9c3ea20ca is the first bad commit commit 8cc5a5f481aeaf6debfa95110c4392d9c3ea20ca HBASE-7172 TestSplitLogManager.testVanishingTaskZNode() fails when run individually and is flaky git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.94@1414975 13f79535-47bb-0310-9956-ffa450edef68 {noformat} Will try again to see if the search ends in the same place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540311#comment-13540311 ] Lars Hofhansl commented on HBASE-3776: -- Patch looks good to me +1 I think we should have this in 0.94 as well. Add Bloom Filter Support to HFileOutputFormat - Key: HBASE-3776 URL: https://issues.apache.org/jira/browse/HBASE-3776 Project: HBase Issue Type: Sub-task Components: mapreduce Reporter: Nicolas Spiegelberg Assignee: Anoop Sam John Priority: Critical Labels: hbase Fix For: 0.96.0 Attachments: HBASE-3776.patch, HBASE-3776_v2.patch Add Bloom Filter support for bulk imports. Lacking a bloom filter, even on a single imported file, can cause perf degradation. Since we now set our compression type based on the HBase CF configuration, it would be good to follow this path for the bloom filter addition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5664) CP hooks in Scan flow for fast forward when filter filters out a row
[ https://issues.apache.org/jira/browse/HBASE-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540312#comment-13540312 ] Anoop Sam John commented on HBASE-5664: --- This is needed in the Secondary index solution The scenario is as below Having 2 conditions in FilterList with MUST_PASS_ALL One SCVF on column cf1:c1 and another scvf on column cf1:c2 Suppose there is only one index on column cf1:c1 Now based on the data got from the index table we will seek to the exact row where cf1:c1=?. But dont know what is the value of cf1:c2 there in that row. Now the second filter may filter out this row. As per the logic in HRegion.nextInternal() it will start a loop there and taking next row onwards till finding a row satisfying both conditions Now as the condition in FL is MUST_PASS_ALL, there is no point in continuing with all the rows. The next row satisfying the condition on column cf1:c1 , we can know from the index data and so we can make a seek to that row directly and check that. Now there is no way to do so. It will be good to provide a CP hook for this so that within that hook I can reseek to the exact row key. CP hooks in Scan flow for fast forward when filter filters out a row Key: HBASE-5664 URL: https://issues.apache.org/jira/browse/HBASE-5664 Project: HBase Issue Type: Improvement Components: Coprocessors Affects Versions: 0.92.1 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.96.0 In HRegion.nextInternal(int limit, String metric) We have while(true) loop so as to fetch a next result which satisfies filter condition. When Filter filters out the current fetched row we call nextRow(byte [] currentRow) before going with the next row. {code} if (results.isEmpty() || filterRow()) { // this seems like a redundant step - we already consumed the row // there're no left overs. // the reasons for calling this method are: // 1. reset the filters. // 2. provide a hook to fast forward the row (used by subclasses) nextRow(currentRow); {code} // 2. provide a hook to fast forward the row (used by subclasses) We can provide same feature of fast forward support for the CP also. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540315#comment-13540315 ] Anoop Sam John commented on HBASE-3776: --- Lars I can give a patch for 94 version also. Will upload tonight.. Add Bloom Filter Support to HFileOutputFormat - Key: HBASE-3776 URL: https://issues.apache.org/jira/browse/HBASE-3776 Project: HBase Issue Type: Sub-task Components: mapreduce Reporter: Nicolas Spiegelberg Assignee: Anoop Sam John Priority: Critical Labels: hbase Fix For: 0.96.0 Attachments: HBASE-3776.patch, HBASE-3776_v2.patch Add Bloom Filter support for bulk imports. Lacking a bloom filter, even on a single imported file, can cause perf degradation. Since we now set our compression type based on the HBase CF configuration, it would be good to follow this path for the bloom filter addition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540318#comment-13540318 ] Ted Yu commented on HBASE-3776: --- Integrated to trunk. Thanks for the patch, Anoop. Thanks for the review, Lars. Add Bloom Filter Support to HFileOutputFormat - Key: HBASE-3776 URL: https://issues.apache.org/jira/browse/HBASE-3776 Project: HBase Issue Type: Sub-task Components: mapreduce Reporter: Nicolas Spiegelberg Assignee: Anoop Sam John Priority: Critical Labels: hbase Fix For: 0.96.0 Attachments: HBASE-3776.patch, HBASE-3776_v2.patch Add Bloom Filter support for bulk imports. Lacking a bloom filter, even on a single imported file, can cause perf degradation. Since we now set our compression type based on the HBase CF configuration, it would be good to follow this path for the bloom filter addition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540322#comment-13540322 ] Hudson commented on HBASE-3776: --- Integrated in HBase-TRUNK #3664 (See [https://builds.apache.org/job/HBase-TRUNK/3664/]) HBASE-3776 Add Bloom Filter Support to HFileOutputFormat (Anoop Sam John) (Revision 1426406) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java Add Bloom Filter Support to HFileOutputFormat - Key: HBASE-3776 URL: https://issues.apache.org/jira/browse/HBASE-3776 Project: HBase Issue Type: Sub-task Components: mapreduce Reporter: Nicolas Spiegelberg Assignee: Anoop Sam John Priority: Critical Labels: hbase Fix For: 0.96.0 Attachments: HBASE-3776.patch, HBASE-3776_v2.patch Add Bloom Filter support for bulk imports. Lacking a bloom filter, even on a single imported file, can cause perf degradation. Since we now set our compression type based on the HBase CF configuration, it would be good to follow this path for the bloom filter addition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7446) Make the reference guide utf8 rather than 8859
[ https://issues.apache.org/jira/browse/HBASE-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540323#comment-13540323 ] Hudson commented on HBASE-7446: --- Integrated in HBase-TRUNK #3664 (See [https://builds.apache.org/job/HBase-TRUNK/3664/]) hbase-7446 Make the reference guide utf8 rather than 8859 (Revision 1426400) Result = FAILURE Make the reference guide utf8 rather than 8859 -- Key: HBASE-7446 URL: https://issues.apache.org/jira/browse/HBASE-7446 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.96.0 Attachments: utf8.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7448) Add tags to Cell API
Andrew Purtell created HBASE-7448: - Summary: Add tags to Cell API Key: HBASE-7448 URL: https://issues.apache.org/jira/browse/HBASE-7448 Project: HBase Issue Type: Task Affects Versions: 0.96.0 Reporter: Andrew Purtell See https://issues.apache.org/jira/browse/HBASE-7233?focusedCommentId=13531619page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13531619 and nearby comments on HBASE-7233 for background. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-3776: -- Attachment: 3776-trunk.addendum Addendum for trunk that fixes reference to BloomType Add Bloom Filter Support to HFileOutputFormat - Key: HBASE-3776 URL: https://issues.apache.org/jira/browse/HBASE-3776 Project: HBase Issue Type: Sub-task Components: mapreduce Reporter: Nicolas Spiegelberg Assignee: Anoop Sam John Priority: Critical Labels: hbase Fix For: 0.96.0 Attachments: 3776-trunk.addendum, HBASE-3776.patch, HBASE-3776_v2.patch Add Bloom Filter support for bulk imports. Lacking a bloom filter, even on a single imported file, can cause perf degradation. Since we now set our compression type based on the HBase CF configuration, it would be good to follow this path for the bloom filter addition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7224) Remove references to Writable in the ipc package
[ https://issues.apache.org/jira/browse/HBASE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540332#comment-13540332 ] stack commented on HBASE-7224: -- Any chance of a review for this blocker against 0.96 issue? Remove references to Writable in the ipc package Key: HBASE-7224 URL: https://issues.apache.org/jira/browse/HBASE-7224 Project: HBase Issue Type: Sub-task Components: IPC/RPC, Protobufs Reporter: Devaraj Das Assignee: stack Priority: Blocker Fix For: 0.96.0 Attachments: 7224.txt, 7224v2.txt, 7224v3.txt, 7224v4.txt, 7224v4.txt, 7224v4.txt, purge_more_writables.txt I see references to Writable in the ipc package, most notably in the Invocation class. This class is not being used that much in the core ipc package but used in the coprocessor protocol implementations (there are some coprocessor protocols that are Writable based still). This jira is to track removing those references and the Invocation class (once HBASE-6895 is resolved). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7448) Add tags to Cell API
[ https://issues.apache.org/jira/browse/HBASE-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-7448: -- Attachment: 7448_todo.txt Attached as 7448_todo.txt just an idea for the interface, implementation TODO. Moves Pair from hbase-server to hbase-common. Add tags to Cell API Key: HBASE-7448 URL: https://issues.apache.org/jira/browse/HBASE-7448 Project: HBase Issue Type: Task Affects Versions: 0.96.0 Reporter: Andrew Purtell Attachments: 7448_todo.txt See https://issues.apache.org/jira/browse/HBASE-7233?focusedCommentId=13531619page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13531619 and nearby comments on HBASE-7233 for background. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5664) CP hooks in Scan flow for fast forward when filter filters out a row
[ https://issues.apache.org/jira/browse/HBASE-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540334#comment-13540334 ] Lars Hofhansl commented on HBASE-5664: -- As long as the reseek is optional it's fine, I think. A (re)seek() is potentially *much* slower than just a next(). So when the next row is only a few next()'s away replacing that with (re)seek() can have disastrous performance effects (as seen in HBASE-6577). CP hooks in Scan flow for fast forward when filter filters out a row Key: HBASE-5664 URL: https://issues.apache.org/jira/browse/HBASE-5664 Project: HBase Issue Type: Improvement Components: Coprocessors Affects Versions: 0.92.1 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.96.0 In HRegion.nextInternal(int limit, String metric) We have while(true) loop so as to fetch a next result which satisfies filter condition. When Filter filters out the current fetched row we call nextRow(byte [] currentRow) before going with the next row. {code} if (results.isEmpty() || filterRow()) { // this seems like a redundant step - we already consumed the row // there're no left overs. // the reasons for calling this method are: // 1. reset the filters. // 2. provide a hook to fast forward the row (used by subclasses) nextRow(currentRow); {code} // 2. provide a hook to fast forward the row (used by subclasses) We can provide same feature of fast forward support for the CP also. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-3776: - Status: Open (was: Patch Available) Add Bloom Filter Support to HFileOutputFormat - Key: HBASE-3776 URL: https://issues.apache.org/jira/browse/HBASE-3776 Project: HBase Issue Type: Sub-task Components: mapreduce Reporter: Nicolas Spiegelberg Assignee: Anoop Sam John Priority: Critical Labels: hbase Fix For: 0.96.0 Attachments: 3776-trunk.addendum, HBASE-3776.patch, HBASE-3776_v2.patch Add Bloom Filter support for bulk imports. Lacking a bloom filter, even on a single imported file, can cause perf degradation. Since we now set our compression type based on the HBase CF configuration, it would be good to follow this path for the bloom filter addition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-3776: - Attachment: 3776-0.94.txt Here's a 0.94 patch. Add Bloom Filter Support to HFileOutputFormat - Key: HBASE-3776 URL: https://issues.apache.org/jira/browse/HBASE-3776 Project: HBase Issue Type: Sub-task Components: mapreduce Reporter: Nicolas Spiegelberg Assignee: Anoop Sam John Priority: Critical Labels: hbase Fix For: 0.96.0 Attachments: 3776-0.94.txt, 3776-trunk.addendum, HBASE-3776.patch, HBASE-3776_v2.patch Add Bloom Filter support for bulk imports. Lacking a bloom filter, even on a single imported file, can cause perf degradation. Since we now set our compression type based on the HBase CF configuration, it would be good to follow this path for the bloom filter addition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7224) Remove references to Writable in the ipc package
[ https://issues.apache.org/jira/browse/HBASE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540336#comment-13540336 ] Devaraj Das commented on HBASE-7224: I'll make a pass over this soon. Remove references to Writable in the ipc package Key: HBASE-7224 URL: https://issues.apache.org/jira/browse/HBASE-7224 Project: HBase Issue Type: Sub-task Components: IPC/RPC, Protobufs Reporter: Devaraj Das Assignee: stack Priority: Blocker Fix For: 0.96.0 Attachments: 7224.txt, 7224v2.txt, 7224v3.txt, 7224v4.txt, 7224v4.txt, 7224v4.txt, purge_more_writables.txt I see references to Writable in the ipc package, most notably in the Invocation class. This class is not being used that much in the core ipc package but used in the coprocessor protocol implementations (there are some coprocessor protocols that are Writable based still). This jira is to track removing those references and the Invocation class (once HBASE-6895 is resolved). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3776) Add Bloom Filter Support to HFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540338#comment-13540338 ] Lars Hofhansl commented on HBASE-3776: -- Opps @Anop. Just saw you latest comment now. Anyway, please review the 0.94 patch I attached. Add Bloom Filter Support to HFileOutputFormat - Key: HBASE-3776 URL: https://issues.apache.org/jira/browse/HBASE-3776 Project: HBase Issue Type: Sub-task Components: mapreduce Reporter: Nicolas Spiegelberg Assignee: Anoop Sam John Priority: Critical Labels: hbase Fix For: 0.96.0 Attachments: 3776-0.94.txt, 3776-trunk.addendum, HBASE-3776.patch, HBASE-3776_v2.patch Add Bloom Filter support for bulk imports. Lacking a bloom filter, even on a single imported file, can cause perf degradation. Since we now set our compression type based on the HBase CF configuration, it would be good to follow this path for the bloom filter addition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7224) Remove references to Writable in the ipc package
[ https://issues.apache.org/jira/browse/HBASE-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540343#comment-13540343 ] Ted Yu commented on HBASE-7224: --- I put some comments on review board. Remove references to Writable in the ipc package Key: HBASE-7224 URL: https://issues.apache.org/jira/browse/HBASE-7224 Project: HBase Issue Type: Sub-task Components: IPC/RPC, Protobufs Reporter: Devaraj Das Assignee: stack Priority: Blocker Fix For: 0.96.0 Attachments: 7224.txt, 7224v2.txt, 7224v3.txt, 7224v4.txt, 7224v4.txt, 7224v4.txt, purge_more_writables.txt I see references to Writable in the ipc package, most notably in the Invocation class. This class is not being used that much in the core ipc package but used in the coprocessor protocol implementations (there are some coprocessor protocols that are Writable based still). This jira is to track removing those references and the Invocation class (once HBASE-6895 is resolved). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7447) Occasional failures of TestSplitLogWorker#testAcquireTaskAtStartup
[ https://issues.apache.org/jira/browse/HBASE-7447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540345#comment-13540345 ] Himanshu Vashishtha commented on HBASE-7447: What's the stack trace? HBASE-6528 might be related. Occasional failures of TestSplitLogWorker#testAcquireTaskAtStartup -- Key: HBASE-7447 URL: https://issues.apache.org/jira/browse/HBASE-7447 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.4 Reporter: Andrew Purtell I see occasional failures (1 out of ~20) on branch 0.94 of TestSplitLogWorker#testAcquireTaskAtStartup {noformat} Running org.apache.hadoop.hbase.regionserver.TestSplitLogWorker Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.737 sec FAILURE! Failed tests: testAcquireTaskAtStartup(org.apache.hadoop.hbase.regionserver.TestSplitLogWorker): ctr=0, oldval=0, newval=1 {noformat} Bisecting commits on 0.94 branch, using (up to) 50 repetitions to include or exclude, I get: {noformat} 8cc5a5f481aeaf6debfa95110c4392d9c3ea20ca is the first bad commit commit 8cc5a5f481aeaf6debfa95110c4392d9c3ea20ca HBASE-7172 TestSplitLogManager.testVanishingTaskZNode() fails when run individually and is flaky git-svn-id: https://svn.apache.org/repos/asf/hbase/branches/0.94@1414975 13f79535-47bb-0310-9956-ffa450edef68 {noformat} Will try again to see if the search ends in the same place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira