[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16367987#comment-16367987 ] Hudson commented on HBASE-19988: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4596 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4596/]) HBASE-19988 Fixed chatty log from HRegion#lockRowsAndBuildMiniBatch() (stack: rev 9497eac4e01a12acfa6660b5b16e73a45bcf5707) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch, > hbase-19988.master.001.patch, hbase-19988.master.002.patch, > hbase-19988.master.002.patch, hbase-19988.master.002.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366816#comment-16366816 ] Hadoop QA commented on HBASE-19988: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 54s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 58s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 11s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 15s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 10s{color} | {color:red} The patch causes 10 errors with Hadoop v2.6.5. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 7s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 15s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green}102m 10s{color} | {color:green} hbase-server in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}129m 16s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19988 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910885/hbase-19988.master.002.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 9273fee5d9dc 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 40f8d20cf7 | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/11550/testReport/ | | Max. process+thread count | 4965 (v
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366298#comment-16366298 ] Umesh Agashe commented on HBASE-19988: -- hmm... failures doesn't seem to be related to the changes. 4 runs, 4 different tests failed each time: hadoop.hbase.replication.TestReplicationDroppedTables hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures hadoop.hbase.quotas.TestQuotaThrottle hadoop.hbase.TestJMXListener > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch, > hbase-19988.master.001.patch, hbase-19988.master.002.patch, > hbase-19988.master.002.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366293#comment-16366293 ] Hadoop QA commented on HBASE-19988: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 18s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 2s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 33s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 20s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 27s{color} | {color:red} The patch causes 10 errors with Hadoop v2.6.5. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 36s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 52s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 20s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}133m 38s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.TestJMXListener | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19988 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910789/hbase-19988.master.002.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux fda3c6d43cd3 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 01a2869363 | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | unit | https://builds.apache.org/job/PreCommit-HBA
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366229#comment-16366229 ] Appy commented on HBASE-19988: -- Yes, sir, am good with the patch. > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch, > hbase-19988.master.001.patch, hbase-19988.master.002.patch, > hbase-19988.master.002.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366156#comment-16366156 ] stack commented on HBASE-19988: --- Retry. The Quota failure is not in https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests-branch2.0/lastSuccessfulBuild/artifact/dashboard.html but it has failed for me on local builds. Lets see if issue. Hadoop checks are the thrift schema download failure. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:2.5.2:install (default-install) on project hbase-thrift: Failed to install metadata org.apache.hbase:hbase-thrift:3.0.0-SNAPSHOT/maven-metadata.xml: Could not parse metadata /home/jenkins/.m2/repository/org/apache/hbase/hbase-thrift/3.0.0-SNAPSHOT/maven-metadata-local.xml: in epilog non whitespace content is not allowed but got / (position: END_TAG seen ...\n/... @25:2) -> [Help 1] [~appy] you good w/ this? > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch, > hbase-19988.master.001.patch, hbase-19988.master.002.patch, > hbase-19988.master.002.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364987#comment-16364987 ] Hadoop QA commented on HBASE-19988: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 52s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 56s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 15s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 14s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 14s{color} | {color:red} The patch causes 10 errors with Hadoop v2.6.5. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 12s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 33s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}106m 18s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}134m 10s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.quotas.TestQuotaThrottle | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19988 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910633/hbase-19988.master.002.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 7d5ff36ada8c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / d0f2d18ca7 | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | unit | https://builds.apache.org/job/PreC
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364827#comment-16364827 ] Umesh Agashe commented on HBASE-19988: -- Thanks [~appy] for digging into the code and posting your thoughts, comments. {quote}Currently the comment says "// We will retry when other exceptions, but we should stop if we timeout ." Should be updated with reasons why we break out for each type. {quote} I agree, I have modified the comment and uploaded the new patch. Please see patch 002. However I prefer current exception handling over suggested as the general guideline is to handle specific exception first and then generic exceptions. As for the log message, getRowLockInternal() itself logs following warning message: {code:java} 2018-02-10 04:24:25,503 WARN [PutThread] regionserver.HRegion(5636): Thread interrupted waiting for lock on row: row0{code} I think this message has details required for debugging without being too chatty. Additionally, with run time type checking code will look like: {code:java} if (isAtomic() or ioe instanceof IIOE or ioe instanceof TIOE) { throw ioe; }{code} As you can see, readability will suffer, if we identify additional exception/ condition for retrying the operation in future. > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch, > hbase-19988.master.001.patch, hbase-19988.master.002.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363378#comment-16363378 ] Appy commented on HBASE-19988: -- Sorry, i don't have time to dig in and come up with a better understanding of handling InterruptedException when processing requests. In this case, since IE was already being converted to IIOE, that means any other operation would have been handling it like IOException, which means cancel the operation. Going by that logic, and status quo bias (that it's already IIOE), i think it might be fine to do this. However, I think it'll be better to handle it as part of IOException by doing {code} if (isAtomic() or ioe instanceof IIOE) { throw ioe; } {code} because it'll log a good warning. Maybe move TimeoutIOException there too. Currently the comment says "// We will retry when other exceptions, but we should stop if we timeout ." Should be updated with reasons why we break out for each type. Let's not leave things in more dismay for future onlookers (why these two? why not others? etc etc). They shouldn't have to spend the time we already did, else our effort is wasted. > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch, > hbase-19988.master.001.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363338#comment-16363338 ] stack commented on HBASE-19988: --- bq. Been reading around code for last 30 min, i honestly have no idea how are we supposed to interpret InterruptedException. IE handling is erratic. Some code lines are non-interruptible (HDFS, client retries...). Generally, if you have an IE and don't know what to do w/ it, do clean up, set interrupt on thread and rethrow. A good project would be going though the codebase throwing IEs to see what happens. > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch, > hbase-19988.master.001.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363330#comment-16363330 ] Umesh Agashe commented on HBASE-19988: -- As surefire is able to interrupt tests suggests that InterruptedException is not ignored always/ from everywhere. > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch, > hbase-19988.master.001.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363326#comment-16363326 ] Appy commented on HBASE-19988: -- Been reading around code for last 30 min, i honestly have no idea how are we supposed to interpret InterruptedException. > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch, > hbase-19988.master.001.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363228#comment-16363228 ] Hadoop QA commented on HBASE-19988: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 16s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 7s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 46s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 38s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 18m 41s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5 2.7.4 or 3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 31s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}142m 56s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19988 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910445/hbase-19988.master.001.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux d6a29d121b63 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 39e191e559 | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/11512/artifact/patchprocess/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/11512/testReport/ | | Max. process+thread count | 5346 (vs. ulimit of 1) | | modules | C: hbase-server U: hbase
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363052#comment-16363052 ] stack commented on HBASE-19988: --- Retry > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch, > hbase-19988.master.001.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363051#comment-16363051 ] Umesh Agashe commented on HBASE-19988: -- Thanks [~stack]! Lets wait for what [~appy] has to say on this. > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch, > hbase-19988.master.001.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363048#comment-16363048 ] stack commented on HBASE-19988: --- Thanks for explanation. +1 on patch then. > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16362989#comment-16362989 ] Umesh Agashe commented on HBASE-19988: -- It was logging following exception... several times! {code:java} 2018-02-10 04:24:25,503 WARN [PutThread] regionserver.HRegion(5636): Thread interrupted waiting for lock on row: row0 2018-02-10 04:24:25,503 WARN [PutThread] regionserver.HRegion$BatchOperation(3173): Failed getting lock, row=row0 java.io.InterruptedIOException at org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5637) at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.lockRowsAndBuildMiniBatch(HRegion.java:3168) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:3837) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3810) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3741) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3732) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3746) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4074) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:2925) at org.apache.hadoop.hbase.regionserver.TestHRegion$PutThread.run(TestHRegion.java:3891) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326) at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:871) at org.apache.hadoop.hbase.regionserver.HRegion.getRowLockInternal(HRegion.java:5621) ... 9 more{code} There is a loop in the write batch path: {code:java} while (!batchOp.isDone()) { doMiniBatchMutate(batchOp); }{code} This loop essentially, tries to acquire locks on as many rows in a batch as possible and creates a mini-batch of those rows to write. Next time, locks are acquired from last row (row for which previous iteration failed to acquire a lock) on till the entire batch is written. The operation was aborted/ stopped only on Timeout exception. All other exceptions were logged and ignored to resume creating and writing mini-batches for an input batch. In this particular case, getRowLockInternal() used to fail with exception InterruptedIOException caused by surefire (possibly due to test timeout). This exception was ignored to proceed with write operation containing locked rows so far. This was causing continuous calls to doMinibatchMutate() in a loop, filling up the logs. > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361795#comment-16361795 ] Hadoop QA commented on HBASE-19988: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s{color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 53s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 58s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m 11s{color} | {color:green} branch has no errors when building our shaded downstream artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m 33s{color} | {color:green} patch has no errors when building our shaded downstream artifacts. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 32s{color} | {color:red} The patch causes 10 errors with Hadoop v2.6.5. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 29s{color} | {color:red} The patch causes 10 errors with Hadoop v2.7.4. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 38s{color} | {color:red} The patch causes 10 errors with Hadoop v3.0.0. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 39s{color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}125m 19s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.replication.TestReplicationDroppedTables | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19988 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12910308/hbase-19988.master.001.patch | | Optional Tests | asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile | | uname | Linux 1f75f120fc96 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 7cc239fb5a | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | Default Java | 1.8.0_151 | | unit | https://builds.apa
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361712#comment-16361712 ] stack commented on HBASE-19988: --- What was it logging? > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19988) HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while waiting for a row lock
[ https://issues.apache.org/jira/browse/HBASE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361707#comment-16361707 ] Umesh Agashe commented on HBASE-19988: -- Handling InterruptedIOException same as Timeout. Re-throwing without logging. Message is logged once from getRowLockInternal(). > HRegion#lockRowsAndBuildMiniBatch() is too chatty when interrupted while > waiting for a row lock > --- > > Key: HBASE-19988 > URL: https://issues.apache.org/jira/browse/HBASE-19988 > Project: HBase > Issue Type: Improvement > Components: amv2 >Affects Versions: 2.0.0-beta-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Minor > Fix For: 2.0.0-beta-2 > > Attachments: hbase-19988.master.001.patch > > > See HBASE-19970, TestHRegionWithInMemoryFlush created 4.2g log file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)