[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13543799#comment-13543799 ] Hudson commented on HBASE-6372: --- Integrated in HBase-TRUNK #3695 (See [https://builds.apache.org/job/HBase-TRUNK/3695/]) HBASE-6694 Test scanner batching in export job feature from HBASE-6372 (Alexander Alten-Lorenz) (Revision 1428772) Result = FAILURE jmhsieh : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Export.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.94.2, 0.96.0 Reporter: Lars George Assignee: Shengsheng Huang Priority: Minor Labels: newbie Fix For: 0.96.0 Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch, HBASE-6372.4.patch, HBASE-6372.5.patch, HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13543832#comment-13543832 ] Hudson commented on HBASE-6372: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #329 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/329/]) HBASE-6694 Test scanner batching in export job feature from HBASE-6372 (Alexander Alten-Lorenz) (Revision 1428772) Result = FAILURE jmhsieh : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Export.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.94.2, 0.96.0 Reporter: Lars George Assignee: Shengsheng Huang Priority: Minor Labels: newbie Fix For: 0.96.0 Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch, HBASE-6372.4.patch, HBASE-6372.5.patch, HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13439799#comment-13439799 ] Hadoop QA commented on HBASE-6372: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542014/HBASE-6372.5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2652//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2652//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2652//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2652//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2652//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2652//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2652//console This message is automatically generated. Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Assignee: Shengsheng Huang Priority: Minor Labels: newbie Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch, HBASE-6372.4.patch, HBASE-6372.5.patch, HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13434458#comment-13434458 ] Alexander Alten-Lorenz commented on HBASE-6372: --- [INFO] Reactor Summary: [INFO] [INFO] HBase . SUCCESS [47.617s] [INFO] HBase - Common SUCCESS [1.882s] [INFO] HBase - Hadoop Compatibility .. SUCCESS [0.744s] [INFO] HBase - Hadoop One Compatibility .. SUCCESS [0.858s] [INFO] HBase - Server SUCCESS [14.455s] [INFO] HBase - Hadoop Two Compatibility .. SUCCESS [4.098s] [INFO] HBase - Integration Tests . SUCCESS [1.408s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 1:13.508s [INFO] Finished at: Tue Aug 14 12:47:05 PDT 2012 [INFO] Final Memory: 42M/184M Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Assignee: Shengsheng Huang Priority: Minor Labels: newbie Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch, HBASE-6372.4.patch, HBASE-6372.5.patch, HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13433250#comment-13433250 ] Alexander Alten-Lorenz commented on HBASE-6372: --- Lars, stack, some thoughts? Would be great when we can commit the patch soon. In my personal HBase build the patch was working. thanks, Alex Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Assignee: Shengsheng Huang Priority: Minor Labels: newbie Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch, HBASE-6372.4.patch, HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13433257#comment-13433257 ] Zhihong Ted Yu commented on HBASE-6372: --- @Alex: There was some hanging test according to https://builds.apache.org/job/PreCommit-HBASE-Build/2512/console Maybe you can submit your patch again to see if tests pass. Also, do you want to address Shengsheng's comment above ? Thanks Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Assignee: Shengsheng Huang Priority: Minor Labels: newbie Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch, HBASE-6372.4.patch, HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13433274#comment-13433274 ] Alexander Alten-Lorenz commented on HBASE-6372: --- Yes, I'll address these too. Patch will follow. Thanks, Alex Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Assignee: Shengsheng Huang Priority: Minor Labels: newbie Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch, HBASE-6372.4.patch, HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429014#comment-13429014 ] Alexander Alten-Lorenz commented on HBASE-6372: --- Hm, these failures can be ignored, right? I was testing with trunk and got no errors. Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Assignee: Shengsheng Huang Priority: Minor Labels: newbie Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch, HBASE-6372.4.patch, HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[
https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429020#comment-13429020
]
Shengsheng Huang commented on HBASE-6372:
-
It seems the current Scan#setBatch function may throw exception. Details as
below:
{quote}
public void setBatch(int batch) {
if (this.hasFilter() this.filter.hasFilterRow()) {
throw new IncompatibleFilterException(
Cannot set batch on a scan using a filter +
that returns true for filter.hasFilterRow);
}
this.batch = batch;
}
{quote}
Don't we need to catch that exception in this patch?
Add scanner batching to Export job
--
Key: HBASE-6372
URL: https://issues.apache.org/jira/browse/HBASE-6372
Project: HBase
Issue Type: Improvement
Components: mapreduce
Affects Versions: 0.96.0, 0.94.2
Reporter: Lars George
Assignee: Shengsheng Huang
Priority: Minor
Labels: newbie
Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch,
HBASE-6372.4.patch, HBASE-6372.patch
When a single row is too large for the RS heap then an OOME can take out the
entire RS. Setting scanner batching in custom scans helps avoiding this
scenario, but for the supplied Export job this is not set.
Similar to HBASE-3421 we can set the batching to a low number - or if needed
make it a command line option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13428816#comment-13428816 ] Hadoop QA commented on HBASE-6372: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539180/HBASE-6372.4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.TestFullLogReconstruction Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2511//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2511//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2511//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2511//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2511//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2511//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2511//console This message is automatically generated. Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Assignee: Shengsheng Huang Priority: Minor Labels: newbie Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch, HBASE-6372.4.patch, HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13428820#comment-13428820 ] Hadoop QA commented on HBASE-6372: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539193/HBASE-6372.4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2512//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2512//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2512//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2512//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2512//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2512//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2512//console This message is automatically generated. Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Assignee: Shengsheng Huang Priority: Minor Labels: newbie Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch, HBASE-6372.4.patch, HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13428672#comment-13428672 ] Lars George commented on HBASE-6372: I meant hbase.client.scanner.batch above. Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Assignee: Shengsheng Huang Priority: Minor Labels: newbie Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch, HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13428804#comment-13428804 ] Alexander Alten-Lorenz commented on HBASE-6372: --- https://reviews.apache.org/r/6388/ Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Assignee: Shengsheng Huang Priority: Minor Labels: newbie Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch, HBASE-6372.4.patch, HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13427880#comment-13427880 ] Hadoop QA commented on HBASE-6372: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539000/HBASE-6372.3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient org.apache.hadoop.hbase.regionserver.TestAtomicOperation Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2497//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2497//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2497//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2497//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2497//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2497//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2497//console This message is automatically generated. Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Assignee: Shengsheng Huang Priority: Minor Labels: newbie Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch, HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[
https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13427952#comment-13427952
]
Lars George commented on HBASE-6372:
A few comments on the patch:
{code}
+ final static String EXPORT_BATCHING = hbase.mapreduce.export.batch;
...
+-Dhbase.client.scanner.caching=100\n
{code}
I see that you followed the config key naming already in use where you declared
your new key. But looking at the other keys being used, they are all over the
place. The one for scanner caching - which is the closest to what you are
adding. I suggest we follow the same rules, i.e. name it
hbase.mapreduce.export.batch.
{code}
+ try {
+s.setBatch(batching);
+ } catch (Exception e) {
+LOG.error(Batching is not set because : +e.toString());
+ }
{code}
Why wrap the setBatch() in a try/except? None of the filter being used are of
the kind that trigger the runtime exception. We can add the try/catch later if
ever needed?
{code}
+int batching = conf.getInt(EXPORT_BATCHING,-1);
{code}
Minor not, there should a space between the command the value.
{code}
+LOG.error(Batching is not set because : +e.toString());
{code}
Same minor nit, no spaces between the string concatenation.
{code}
+System.err.println(For very wide rows consider set scan batching
properties as below:\n
{code}
Maybe rephrase a bit? For example: For tables with very wide rows consider
setting the batch size as below:.
Add scanner batching to Export job
--
Key: HBASE-6372
URL: https://issues.apache.org/jira/browse/HBASE-6372
Project: HBase
Issue Type: Improvement
Components: mapreduce
Affects Versions: 0.96.0, 0.94.2
Reporter: Lars George
Assignee: Shengsheng Huang
Priority: Minor
Labels: newbie
Attachments: HBASE-6372.2.patch, HBASE-6372.3.patch, HBASE-6372.patch
When a single row is too large for the RS heap then an OOME can take out the
entire RS. Setting scanner batching in custom scans helps avoiding this
scenario, but for the supplied Export job this is not set.
Similar to HBASE-3421 we can set the batching to a low number - or if needed
make it a command line option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[
https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13427791#comment-13427791
]
Shengsheng Huang commented on HBASE-6372:
-
{quote}
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestAtomicOperation
{quote}
This unit test passed at my local deployment. And I don't think the patch has
anything to do with this unit test.
Could somebody kindly help confirm why the unit test fail in Hadoop QA?
Thanks very much.
Add scanner batching to Export job
--
Key: HBASE-6372
URL: https://issues.apache.org/jira/browse/HBASE-6372
Project: HBase
Issue Type: Improvement
Components: mapreduce
Affects Versions: 0.96.0, 0.94.2
Reporter: Lars George
Assignee: Shengsheng Huang
Priority: Minor
Labels: newbie
Attachments: HBASE-6372.2.patch, HBASE-6372.patch
When a single row is too large for the RS heap then an OOME can take out the
entire RS. Setting scanner batching in custom scans helps avoiding this
scenario, but for the supplied Export job this is not set.
Similar to HBASE-3421 we can set the batching to a low number - or if needed
make it a command line option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[
https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13427830#comment-13427830
]
Zhihong Ted Yu commented on HBASE-6372:
---
Lars did some work in HBASE-5659 to improve TestAtomicOperation
From
https://builds.apache.org/job/PreCommit-HBASE-Build/2459//testReport/org.apache.hadoop.hbase.regionserver/TestAtomicOperation/testMultiRowMutationMultiThreads/:
{code}
2012-07-31 09:22:36,252 DEBUG [Thread-124]
regionserver.TestAtomicOperation$2(398):
[rowA/colfamily11:qual1/4069/Put/vlen=6/ts=4278,
rowB/colfamily11:qual1/4073/Put/vlen=6/ts=0]
Exception in thread Thread-124 junit.framework.AssertionFailedError at
junit.framework.Assert.fail(Assert.java:48)
at junit.framework.Assert.fail(Assert.java:56)
at
org.apache.hadoop.hbase.regionserver.TestAtomicOperation$2.run(TestAtomicOperation.java:400)
{code}
This corresponds to the following:
{code}
if (r.size() != 1) {
LOG.debug(r);
failures.incrementAndGet();
fail();
{code}
Maybe Lars has better idea about the above.
About the patch:
{code}
++-Dhbase.mapreduce.export.batch=10);
{code}
Please use EXPORT_BATCHING in place of duplicate config parameter name above.
nit:
{code}
+ final static String EXPORT_BATCHING=hbase.mapreduce.export.batch;
{code}
Introduce spaces around = sign.
Add scanner batching to Export job
--
Key: HBASE-6372
URL: https://issues.apache.org/jira/browse/HBASE-6372
Project: HBase
Issue Type: Improvement
Components: mapreduce
Affects Versions: 0.96.0, 0.94.2
Reporter: Lars George
Assignee: Shengsheng Huang
Priority: Minor
Labels: newbie
Attachments: HBASE-6372.2.patch, HBASE-6372.patch
When a single row is too large for the RS heap then an OOME can take out the
entire RS. Setting scanner batching in custom scans helps avoiding this
scenario, but for the supplied Export job this is not set.
Similar to HBASE-3421 we can set the batching to a low number - or if needed
make it a command line option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13425643#comment-13425643 ] Hadoop QA commented on HBASE-6372: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538518/HBASE-6372.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestAtomicOperation Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2459//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2459//console This message is automatically generated. Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Assignee: Shengsheng Huang Priority: Minor Labels: newbie Attachments: HBASE-6372.2.patch, HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[
https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13424925#comment-13424925
]
Jonathan Hsieh commented on HBASE-6372:
---
nit:
can we also make this
{code}
+int batching = Integer.parseInt(conf.get(EXPORT_BATCHING,-1));
{code}
the more canonical:
{code}
+int batching = conf.getInt(EXPORT_BATCHING,-1));
{code}
before we commit?
Add scanner batching to Export job
--
Key: HBASE-6372
URL: https://issues.apache.org/jira/browse/HBASE-6372
Project: HBase
Issue Type: Improvement
Components: mapreduce
Affects Versions: 0.96.0, 0.94.2
Reporter: Lars George
Assignee: Shengsheng Huang
Priority: Minor
Labels: newbie
Attachments: HBASE-6372.patch
When a single row is too large for the RS heap then an OOME can take out the
entire RS. Setting scanner batching in custom scans helps avoiding this
scenario, but for the supplied Export job this is not set.
Similar to HBASE-3421 we can set the batching to a low number - or if needed
make it a command line option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[
https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423698#comment-13423698
]
Shengsheng Huang commented on HBASE-6372:
-
{quote}
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
{quote}
I don't think this patch has something to do with this unit test. This unit
test had passed at my local deployment.
Add scanner batching to Export job
--
Key: HBASE-6372
URL: https://issues.apache.org/jira/browse/HBASE-6372
Project: HBase
Issue Type: Improvement
Components: mapreduce
Affects Versions: 0.96.0, 0.94.2
Reporter: Lars George
Priority: Minor
Labels: newbie
Attachments: HBASE-6372.patch
When a single row is too large for the RS heap then an OOME can take out the
entire RS. Setting scanner batching in custom scans helps avoiding this
scenario, but for the supplied Export job this is not set.
Similar to HBASE-3421 we can set the batching to a low number - or if needed
make it a command line option.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423446#comment-13423446 ] Lars Hofhansl commented on HBASE-6372: -- 0.94 sounds good to me. Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Priority: Minor Labels: newbie Attachments: HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423487#comment-13423487 ] Hadoop QA commented on HBASE-6372: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537954/HBASE-6372.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//console This message is automatically generated. Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Priority: Minor Labels: newbie Attachments: HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
