[jira] [Commented] (HBASE-2247) [performance] Small scans should run like a random-read
[ https://issues.apache.org/jira/browse/HBASE-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461331#comment-13461331 ] LiuLei commented on HBASE-2247: --- Hi stackļ¼ I think We can new smallScan() method in HTable, the method use pread to read data and the method return all data in one hfile block. I think we can let users to decide use smallScan() method or getScanner method. [performance] Small scans should run like a random-read --- Key: HBASE-2247 URL: https://issues.apache.org/jira/browse/HBASE-2247 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack hbase-2180 added pread whenever we do a get -- random-read -- and kept the old sync+position+read for when scanning. In between is the case of small scans. Small scans of 0-100 or so rows where the cells are small will likely fit a single hfile blocksize, especially if its the default 64k. We should recognize small scans and flip to random-read to satisfy (somehow). It'll up the performance a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6869) Update our hadoop-2 to 2.0.1-alpha
[ https://issues.apache.org/jira/browse/HBASE-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6869: - Attachment: 6869.txt Update our hadoop-2 to 2.0.1-alpha -- Key: HBASE-6869 URL: https://issues.apache.org/jira/browse/HBASE-6869 Project: HBase Issue Type: Bug Reporter: stack Attachments: 6869.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6869) Update our hadoop-2 to 2.0.1-alpha
[ https://issues.apache.org/jira/browse/HBASE-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-6869. -- Resolution: Fixed Fix Version/s: 0.96.0 Assignee: stack Update from 2.0.0-alpha to 2.0.1-alpha... committed to trunk. Update our hadoop-2 to 2.0.1-alpha -- Key: HBASE-6869 URL: https://issues.apache.org/jira/browse/HBASE-6869 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.96.0 Attachments: 6869.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6869) Update our hadoop-2 to 2.0.1-alpha
[ https://issues.apache.org/jira/browse/HBASE-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461447#comment-13461447 ] Hudson commented on HBASE-6869: --- Integrated in HBase-TRUNK #3371 (See [https://builds.apache.org/job/HBase-TRUNK/3371/]) HBASE-6869 Update our hadoop-2 to 2.0.1-alpha (Revision 1389071) Result = FAILURE stack : Files : * /hbase/trunk/pom.xml Update our hadoop-2 to 2.0.1-alpha -- Key: HBASE-6869 URL: https://issues.apache.org/jira/browse/HBASE-6869 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.96.0 Attachments: 6869.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
[ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461485#comment-13461485 ] Lars Hofhansl commented on HBASE-6868: -- OK. So can we just document then that dfs.client.read.shortcircuit.skip.checksum should not be enabled globally, and then we only enable it for the nonCheckSumFs? If that does not work I let's revert HBASE-5074 (at least from 0.94). It's not worth it. (Although that will be a big task now, since so much has changed since it got committed.) The 2nd part you mention sounds like another bug in HDFS with local reads. Skip checksum is broke; are we double-checksumming by default? -- Key: HBASE-6868 URL: https://issues.apache.org/jira/browse/HBASE-6868 Project: HBase Issue Type: Bug Components: HFile, wal Affects Versions: 0.94.0, 0.94.1 Reporter: LiuLei Priority: Blocker Fix For: 0.94.3, 0.96.0 The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't need to read the checksum from meta file of HDFS. But HLog file of Hbase don't contain the checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS. We could add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip checksum facility is enabled -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
[ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461494#comment-13461494 ] Lars Hofhansl commented on HBASE-6868: -- We keep going back and forth between different issues. There at least three issues now discussed here: # HLogs are not checksummed when dfs.client.read.shortcircuit.skip.checksum and dfs.client.read.shortcircuit are both true # double checksumming when dfs.client.read.shortcircuit = true and dfs.client.read.shortcircuit.skip.checksum=false # local files lingering with DN is down and file was deleted via DFS. The first two are related. Skip checksum is broke; are we double-checksumming by default? -- Key: HBASE-6868 URL: https://issues.apache.org/jira/browse/HBASE-6868 Project: HBase Issue Type: Bug Components: HFile, wal Affects Versions: 0.94.0, 0.94.1 Reporter: LiuLei Priority: Blocker Fix For: 0.94.3, 0.96.0 The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't need to read the checksum from meta file of HDFS. But HLog file of Hbase don't contain the checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS. We could add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip checksum facility is enabled -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
[ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6868: - Attachment: 6868-0.96-idea.txt So how about this? Check that short circuit is enabled for HBase checksumming. Also disables short circuit checksum for the noChecksumFs. Skip checksum is broke; are we double-checksumming by default? -- Key: HBASE-6868 URL: https://issues.apache.org/jira/browse/HBASE-6868 Project: HBase Issue Type: Bug Components: HFile, wal Affects Versions: 0.94.0, 0.94.1 Reporter: LiuLei Priority: Blocker Fix For: 0.94.3, 0.96.0 The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't need to read the checksum from meta file of HDFS. But HLog file of Hbase don't contain the checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS. We could add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip checksum facility is enabled -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
[ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6868: - Attachment: (was: 6868-0.96-idea.txt) Skip checksum is broke; are we double-checksumming by default? -- Key: HBASE-6868 URL: https://issues.apache.org/jira/browse/HBASE-6868 Project: HBase Issue Type: Bug Components: HFile, wal Affects Versions: 0.94.0, 0.94.1 Reporter: LiuLei Priority: Blocker Fix For: 0.94.3, 0.96.0 The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't need to read the checksum from meta file of HDFS. But HLog file of Hbase don't contain the checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS. We could add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip checksum facility is enabled -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6868) Skip checksum is broke; are we double-checksumming by default?
[ https://issues.apache.org/jira/browse/HBASE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6868: - Attachment: 6868-0.96-idea.txt The right patch, this time. Skip checksum is broke; are we double-checksumming by default? -- Key: HBASE-6868 URL: https://issues.apache.org/jira/browse/HBASE-6868 Project: HBase Issue Type: Bug Components: HFile, wal Affects Versions: 0.94.0, 0.94.1 Reporter: LiuLei Priority: Blocker Fix For: 0.94.3, 0.96.0 Attachments: 6868-0.96-idea.txt The HFile contains checksums for decrease the iops, so when Hbase read HFile , that dont't need to read the checksum from meta file of HDFS. But HLog file of Hbase don't contain the checksum, so when HBase read the HLog, that must read checksum from meta file of HDFS. We could add setSkipChecksum per file to hdfs or we could write checksums into WAL if this skip checksum facility is enabled -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6381) AssignmentManager should use the same logic for clean startup and failover
[ https://issues.apache.org/jira/browse/HBASE-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6381: --- Attachment: trunk-6381_v9.patch AssignmentManager should use the same logic for clean startup and failover -- Key: HBASE-6381 URL: https://issues.apache.org/jira/browse/HBASE-6381 Project: HBase Issue Type: Bug Components: master Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-6381-notes.pdf, hbase-6381.pdf, trunk-6381_v5.patch, trunk-6381_v7.patch, trunk-6381_v8.patch, trunk-6381_v9.patch Currently AssignmentManager handles clean startup and failover very differently. Different logic is mingled together so it is hard to find out which is for which. We should clean it up and share the same logic so that AssignmentManager handles both cases the same way. This way, the code will much easier to understand and maintain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6381) AssignmentManager should use the same logic for clean startup and failover
[ https://issues.apache.org/jira/browse/HBASE-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6381: --- Status: Open (was: Patch Available) AssignmentManager should use the same logic for clean startup and failover -- Key: HBASE-6381 URL: https://issues.apache.org/jira/browse/HBASE-6381 Project: HBase Issue Type: Bug Components: master Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-6381-notes.pdf, hbase-6381.pdf, trunk-6381_v5.patch, trunk-6381_v7.patch, trunk-6381_v8.patch, trunk-6381_v9.patch Currently AssignmentManager handles clean startup and failover very differently. Different logic is mingled together so it is hard to find out which is for which. We should clean it up and share the same logic so that AssignmentManager handles both cases the same way. This way, the code will much easier to understand and maintain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6381) AssignmentManager should use the same logic for clean startup and failover
[ https://issues.apache.org/jira/browse/HBASE-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6381: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) AssignmentManager should use the same logic for clean startup and failover -- Key: HBASE-6381 URL: https://issues.apache.org/jira/browse/HBASE-6381 Project: HBase Issue Type: Bug Components: master Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-6381-notes.pdf, hbase-6381.pdf, trunk-6381_v5.patch, trunk-6381_v7.patch, trunk-6381_v8.patch, trunk-6381_v9.patch Currently AssignmentManager handles clean startup and failover very differently. Different logic is mingled together so it is hard to find out which is for which. We should clean it up and share the same logic so that AssignmentManager handles both cases the same way. This way, the code will much easier to understand and maintain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6381) AssignmentManager should use the same logic for clean startup and failover
[ https://issues.apache.org/jira/browse/HBASE-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461520#comment-13461520 ] Jimmy Xiang commented on HBASE-6381: The latest patch addressed Rajesh's comments. Thanks Ram for reviewing the fix for that. Will commit it tomorrow if no objection. AssignmentManager should use the same logic for clean startup and failover -- Key: HBASE-6381 URL: https://issues.apache.org/jira/browse/HBASE-6381 Project: HBase Issue Type: Bug Components: master Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-6381-notes.pdf, hbase-6381.pdf, trunk-6381_v5.patch, trunk-6381_v7.patch, trunk-6381_v8.patch, trunk-6381_v9.patch Currently AssignmentManager handles clean startup and failover very differently. Different logic is mingled together so it is hard to find out which is for which. We should clean it up and share the same logic so that AssignmentManager handles both cases the same way. This way, the code will much easier to understand and maintain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6071) getRegionServerWithRetires, should log unsuccessful attempts and exceptions.
[ https://issues.apache.org/jira/browse/HBASE-6071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igal Shilman updated HBASE-6071: Attachment: HBASE-6071.v5.patch Attaching a version, that logs unsuccessful attempts only if withRetires() exits abnormally (neither returns a value, nor RetriesExhaustedException is thrown). getRegionServerWithRetires, should log unsuccessful attempts and exceptions. Key: HBASE-6071 URL: https://issues.apache.org/jira/browse/HBASE-6071 Project: HBase Issue Type: Improvement Components: Client, IPC/RPC Affects Versions: 0.92.0, 0.94.0 Reporter: Igal Shilman Priority: Minor Labels: client, ipc Attachments: HBASE-6071.patch, HBASE-6071.v2.patch, HBASE-6071.v3.patch, HBASE-6071.v4.patch, HBASE-6071.v5.patch, HConnectionManager_HBASE-6071-0.90.0.patch, lease-exception.txt HConnectionImplementation.getRegionServerWithRetries might terminate w/ an exception different then a DoNotRetryIOException, thus silently drops exceptions from previous attempts. [~ted_yu] suggested ([here|http://mail-archives.apache.org/mod_mbox/hbase-user/201205.mbox/%3CCAFebPXBq9V9BVdzRTNr-MB3a1Lz78SZj6gvP6On0b%2Bajt9StAg%40mail.gmail.com%3E]) adding a log message inside the catch block describing the exception type and details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6381) AssignmentManager should use the same logic for clean startup and failover
[ https://issues.apache.org/jira/browse/HBASE-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461527#comment-13461527 ] Hadoop QA commented on HBASE-6381: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546226/trunk-6381_v9.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 21 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 140 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2924//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2924//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2924//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2924//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2924//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2924//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2924//console This message is automatically generated. AssignmentManager should use the same logic for clean startup and failover -- Key: HBASE-6381 URL: https://issues.apache.org/jira/browse/HBASE-6381 Project: HBase Issue Type: Bug Components: master Reporter: Jimmy Xiang Assignee: Jimmy Xiang Attachments: hbase-6381-notes.pdf, hbase-6381.pdf, trunk-6381_v5.patch, trunk-6381_v7.patch, trunk-6381_v8.patch, trunk-6381_v9.patch Currently AssignmentManager handles clean startup and failover very differently. Different logic is mingled together so it is hard to find out which is for which. We should clean it up and share the same logic so that AssignmentManager handles both cases the same way. This way, the code will much easier to understand and maintain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6071) getRegionServerWithRetires, should log unsuccessful attempts and exceptions.
[ https://issues.apache.org/jira/browse/HBASE-6071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461533#comment-13461533 ] Hadoop QA commented on HBASE-6071: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546227/HBASE-6071.v5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 139 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2925//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2925//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2925//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2925//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2925//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2925//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2925//console This message is automatically generated. getRegionServerWithRetires, should log unsuccessful attempts and exceptions. Key: HBASE-6071 URL: https://issues.apache.org/jira/browse/HBASE-6071 Project: HBase Issue Type: Improvement Components: Client, IPC/RPC Affects Versions: 0.92.0, 0.94.0 Reporter: Igal Shilman Priority: Minor Labels: client, ipc Attachments: HBASE-6071.patch, HBASE-6071.v2.patch, HBASE-6071.v3.patch, HBASE-6071.v4.patch, HBASE-6071.v5.patch, HConnectionManager_HBASE-6071-0.90.0.patch, lease-exception.txt HConnectionImplementation.getRegionServerWithRetries might terminate w/ an exception different then a DoNotRetryIOException, thus silently drops exceptions from previous attempts. [~ted_yu] suggested ([here|http://mail-archives.apache.org/mod_mbox/hbase-user/201205.mbox/%3CCAFebPXBq9V9BVdzRTNr-MB3a1Lz78SZj6gvP6On0b%2Bajt9StAg%40mail.gmail.com%3E]) adding a log message inside the catch block describing the exception type and details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6869) Update our hadoop-2 to 2.0.1-alpha
[ https://issues.apache.org/jira/browse/HBASE-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461538#comment-13461538 ] Hudson commented on HBASE-6869: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #189 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/189/]) HBASE-6869 Update our hadoop-2 to 2.0.1-alpha (Revision 1389071) Result = FAILURE stack : Files : * /hbase/trunk/pom.xml Update our hadoop-2 to 2.0.1-alpha -- Key: HBASE-6869 URL: https://issues.apache.org/jira/browse/HBASE-6869 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.96.0 Attachments: 6869.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6869) Update our hadoop-2 to 2.0.1-alpha
[ https://issues.apache.org/jira/browse/HBASE-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461544#comment-13461544 ] Andrew Purtell commented on HBASE-6869: --- @Lars: There was discussion (on general@ I think) about dropping current branch-2.1.0, doing a branch-2.0.2 instead, and recreating branch-2.1.0 later. Perhaps we should hold off on that a bit? Update our hadoop-2 to 2.0.1-alpha -- Key: HBASE-6869 URL: https://issues.apache.org/jira/browse/HBASE-6869 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.96.0 Attachments: 6869.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2247) [performance] Small scans should run like a random-read
[ https://issues.apache.org/jira/browse/HBASE-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461546#comment-13461546 ] Andrew Purtell commented on HBASE-2247: --- -1 on adding a new API that duplicates another but with one distinction. Make this a Scan option instead. [performance] Small scans should run like a random-read --- Key: HBASE-2247 URL: https://issues.apache.org/jira/browse/HBASE-2247 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack hbase-2180 added pread whenever we do a get -- random-read -- and kept the old sync+position+read for when scanning. In between is the case of small scans. Small scans of 0-100 or so rows where the cells are small will likely fit a single hfile blocksize, especially if its the default 64k. We should recognize small scans and flip to random-read to satisfy (somehow). It'll up the performance a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6852) SchemaMetrics.updateOnCacheHit costs too much while full scanning a table with all of its fields
[
https://issues.apache.org/jira/browse/HBASE-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461559#comment-13461559
]
Cheng Hao commented on HBASE-6852:
--
{quote} Cheng Hao: you said that your dataset size was 600GB, and the total
amount of block cache was presumably much smaller than that, which makes me
think the workload should have been I/O-bound. What was the CPU utilization on
your test? What was the disk throughput?
{quote}
Actually it's the CPU-bound. and the utilization is more than 80%.
I have 4 machines and each machine has 12 disks and 24 CPU cores.
Besides, in order to make it more effective, I have splitted the regions twice,
and then did the major compact, to be sure the data locality. After that, I ran
the data scanning tests base on Hive query like select count() from xxx;
I am also curious if there any overheads of threads/syscalls switching (like
during the IPC). PS: I did set the hbase.client.scanner.caching as 1000;
SchemaMetrics.updateOnCacheHit costs too much while full scanning a table
with all of its fields
Key: HBASE-6852
URL: https://issues.apache.org/jira/browse/HBASE-6852
Project: HBase
Issue Type: Improvement
Components: metrics
Affects Versions: 0.94.0
Reporter: Cheng Hao
Priority: Minor
Labels: performance
Fix For: 0.94.3, 0.96.0
Attachments: onhitcache-trunk.patch
The SchemaMetrics.updateOnCacheHit costs too much while I am doing the full
table scanning.
Here is the top 5 hotspots within regionserver while full scanning a table:
(Sorry for the less-well-format)
CPU: Intel Westmere microarchitecture, speed 2.262e+06 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit
mask of 0x00 (No unit mask) count 500
samples %image name symbol name
---
9844713.4324 14033.jo void
org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory,
boolean)
98447100.000 14033.jo void
org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics.updateOnCacheHit(org.apache.hadoop.hbase.io.hfile.BlockType$BlockCategory,
boolean) [self]
---
45814 6.2510 14033.jo int
org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int,
byte[], int, int)
45814100.000 14033.jo int
org.apache.hadoop.hbase.KeyValue$KeyComparator.compareRows(byte[], int, int,
byte[], int, int) [self]
---
43523 5.9384 14033.jo boolean
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue)
43523100.000 14033.jo boolean
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(org.apache.hadoop.hbase.KeyValue)
[self]
---
42548 5.8054 14033.jo int
org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int,
byte[], int, int)
42548100.000 14033.jo int
org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(byte[], int, int,
byte[], int, int) [self]
---
40572 5.5358 14033.jo int
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[],
int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1
40572100.000 14033.jo int
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.binarySearchNonRootIndex(byte[],
int, int, java.nio.ByteBuffer, org.apache.hadoop.io.RawComparator)~1 [self]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3909) Add dynamic config
[ https://issues.apache.org/jira/browse/HBASE-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461560#comment-13461560 ] Subbu M Iyer commented on HBASE-3909: - Stack: Which RB version should l target my patch on ? Please let me know. thanks Subbu Add dynamic config -- Key: HBASE-3909 URL: https://issues.apache.org/jira/browse/HBASE-3909 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.96.0 Attachments: 3909_090712-2.patch, 3909.v1, 3909-v1.patch, HBase Cluster Config Details.xlsx, patch-v2.patch I'm sure this issue exists already, at least as part of the discussion around making online schema edits possible, but no hard this having its own issue. Ted started a conversation on this topic up on dev and Todd suggested we lookd at how Hadoop did it over in HADOOP-7001 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2247) [performance] Small scans should run like a random-read
[
https://issues.apache.org/jira/browse/HBASE-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461563#comment-13461563
]
binlijin commented on HBASE-2247:
-
or we can take the hint information in the Scan object,
{code}
HRegionInterface:
public long openScanner(final byte [] regionName, final Scan scan).
{code}
[performance] Small scans should run like a random-read
---
Key: HBASE-2247
URL: https://issues.apache.org/jira/browse/HBASE-2247
Project: HBase
Issue Type: Improvement
Components: Performance
Reporter: stack
hbase-2180 added pread whenever we do a get -- random-read -- and kept the
old sync+position+read for when scanning. In between is the case of small
scans. Small scans of 0-100 or so rows where the cells are small will likely
fit a single hfile blocksize, especially if its the default 64k. We should
recognize small scans and flip to random-read to satisfy (somehow). It'll up
the performance a bit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6854) Deletion of SPLITTING node on split rollback should clear the region from RIT
[
https://issues.apache.org/jira/browse/HBASE-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461565#comment-13461565
]
Jieshan Bean commented on HBASE-6854:
-
We also found the same problem. Only 2 minor comments:
1.
{code}
LOG.debug(Ephemeral node deleted. Found in SPLIITING state. + Removing
from RIT
{code}
SPLIITING should be SPLITTING.
2. HBaseAdmin in testShouldClearRITWhenNodeFoundInSplittingState should be
closed in finally block.
Otherwise, I'm +1 on this patch.
Deletion of SPLITTING node on split rollback should clear the region from RIT
-
Key: HBASE-6854
URL: https://issues.apache.org/jira/browse/HBASE-6854
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Fix For: 0.94.3
Attachments: HBASE-6854.patch
If a failure happens in split before OFFLINING_PARENT, we tend to rollback
the split including deleting the znodes created.
On deletion of the RS_ZK_SPLITTING node we are getting a callback but not
remvoving from RIT. We need to remove it from RIT, anyway SSH logic is well
guarded in case the delete event comes due to RS down scenario.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2247) [performance] Small scans should run like a random-read
[ https://issues.apache.org/jira/browse/HBASE-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461575#comment-13461575 ] LiuLei commented on HBASE-2247: --- +1 on adding one option in Scan. [performance] Small scans should run like a random-read --- Key: HBASE-2247 URL: https://issues.apache.org/jira/browse/HBASE-2247 Project: HBase Issue Type: Improvement Components: Performance Reporter: stack hbase-2180 added pread whenever we do a get -- random-read -- and kept the old sync+position+read for when scanning. In between is the case of small scans. Small scans of 0-100 or so rows where the cells are small will likely fit a single hfile blocksize, especially if its the default 64k. We should recognize small scans and flip to random-read to satisfy (somehow). It'll up the performance a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6679) RegionServer aborts due to race between compaction and split
[ https://issues.apache.org/jira/browse/HBASE-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461594#comment-13461594 ] ramkrishna.s.vasudevan commented on HBASE-6679: --- Any updates on this issue? RegionServer aborts due to race between compaction and split Key: HBASE-6679 URL: https://issues.apache.org/jira/browse/HBASE-6679 Project: HBase Issue Type: Bug Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.92.3 Attachments: rs-crash-parallel-compact-split.log In our nightlies, we have seen RS aborts due to compaction and split racing. Original parent file gets deleted after the compaction, and hence, the daughters don't find the parent data file. The RS kills itself when this happens. Will attach a snippet of the relevant RS logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6854) Deletion of SPLITTING node on split rollback should clear the region from RIT
[ https://issues.apache.org/jira/browse/HBASE-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461597#comment-13461597 ] ramkrishna.s.vasudevan commented on HBASE-6854: --- @Jieshan Is it ok if i address these 2 comments on commit ? If not i can prepare another patch from home. Deletion of SPLITTING node on split rollback should clear the region from RIT - Key: HBASE-6854 URL: https://issues.apache.org/jira/browse/HBASE-6854 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Fix For: 0.94.3 Attachments: HBASE-6854.patch If a failure happens in split before OFFLINING_PARENT, we tend to rollback the split including deleting the znodes created. On deletion of the RS_ZK_SPLITTING node we are getting a callback but not remvoving from RIT. We need to remove it from RIT, anyway SSH logic is well guarded in case the delete event comes due to RS down scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6870) HTable#coprocessorExec always scan the whole table
chunhui shen created HBASE-6870: --- Summary: HTable#coprocessorExec always scan the whole table Key: HBASE-6870 URL: https://issues.apache.org/jira/browse/HBASE-6870 Project: HBase Issue Type: Improvement Components: Coprocessors Affects Versions: 0.94.1 Reporter: chunhui shen Assignee: chunhui shen In current logic, HTable#coprocessorExec always scan the whole table, its efficiency is low and will affect the Regionserver carrying .META. under large coprocessorExec requests -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6870) HTable#coprocessorExec always scan the whole table
[ https://issues.apache.org/jira/browse/HBASE-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6870: Attachment: HBASE-6870.patch Instead of scanning .META. for whole table,We could use region location cache HTable#coprocessorExec always scan the whole table --- Key: HBASE-6870 URL: https://issues.apache.org/jira/browse/HBASE-6870 Project: HBase Issue Type: Improvement Components: Coprocessors Affects Versions: 0.94.1 Reporter: chunhui shen Assignee: chunhui shen Attachments: HBASE-6870.patch In current logic, HTable#coprocessorExec always scan the whole table, its efficiency is low and will affect the Regionserver carrying .META. under large coprocessorExec requests -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
