[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies
[ https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349984#comment-16349984 ] Hudson commented on HBASE-19901: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4512 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4512/]) Revert "HBASE-19901 HBASE-19901 Up yetus proclimit on nightlies" (stack: rev 18eec8c1a58574ae3ff529904f677288ff696765) * (edit) dev-support/hbase_nightly_yetus.sh * (edit) dev-support/hbase-personality.sh HBASE-19901 HBASE-19901 Up yetus proclimit on nightlies; REAPPLY TO TEST (stack: rev cb7bfc21da4e2658602a9a2b2080c4d883287dd1) * (edit) dev-support/hbase_nightly_yetus.sh * (edit) dev-support/hbase-personality.sh HBASE-19901 HBASE-19901 Up yetus proclimit on nightlies; AMENDMENT (stack: rev a2bc19aa112b8cd0697845d0825d277ff3a8bcfc) * (edit) dev-support/hbase_nightly_yetus.sh > Up yetus proclimit on nightlies > --- > > Key: HBASE-19901 > URL: https://issues.apache.org/jira/browse/HBASE-19901 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-19901.master.001.patch, > HBASE-19901.master.002.patch > > > We're on 0.7.0 now which enforces limits meant to protect against runaway > processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. > Up our proclimit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies
[ https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349857#comment-16349857 ] stack commented on HBASE-19901: --- I reverted all changes made against this issue, then put back the originals. With DEBUG I see that the hbase_nightly is not picking up defines in hbase-personality... So nightly was failing because docker mem set to ... ? Ditto on proclimit. So hardcoded values in hbase_nightly. Not nice. Nightly seems to be working now though. To be revisited. Here is what is committed currently... commit b2fe241089dddaf65253ad882fbbff36225dafd2 Author: Michael StackDate: Thu Feb 1 20:54:13 2018 -0800 HBASE-19901 HBASE-19901 Up yetus proclimit on nightlies; AMENDMENT hardcode proclimit and docker memlimit in nightly script... diff --git a/dev-support/hbase_nightly_yetus.sh b/dev-support/hbase_nightly_yetus.sh index 651a2e22d7..27d11c8beb 100755 --- a/dev-support/hbase_nightly_yetus.sh +++ b/dev-support/hbase_nightly_yetus.sh @@ -67,8 +67,9 @@ YETUS_ARGS=("--whitespace-tabs-ignore-list=${WHITESPACE_IGNORE_LIST}" "${YETUS_A YETUS_ARGS=("--sentinel" "${YETUS_ARGS[@]}") YETUS_ARGS=("--branch=${BRANCH_NAME}" "${YETUS_ARGS[@]}") YETUS_ARGS=("--tests-filter=${TESTS_FILTER}" "${YETUS_ARGS[@]}") -YETUS_ARGS=("--proclimit=${PROCLIMIT}" "${YETUS_ARGS[@]}") -YETUS_ARGS=("--dockermemlimit=${DOCKERMEMLIMIT}" "${YETUS_ARGS[@]}") +# Why are these not being picked up from hbase-personality? +YETUS_ARGS=("--proclimit=1" "${YETUS_ARGS[@]}") +YETUS_ARGS=("--dockermemlimit=20g" "${YETUS_ARGS[@]}") # Currently, flaky list is calculated only for master branch. UNDERSCORED_BRANCH_NAME=$(echo ${BRANCH_NAME} | tr '.-' '_') The defines in hbase-personality: # Yetus 0.7.0 enforces limits. Default proclimit is 1000. # Up it. See HBASE-19902 for how we arrived at this number. PROCLIMIT=1 # Set docker container to run with 20g. Default is 4g in yetus. # See HBASE-19902 for how we arrived at 20g. DOCKERMEMLIMIT=20g ... are what are not being picked up. I traced download of hbase-personality as personality.sh... but don't see how it picked up (though yetus describes picking up these globals...) TODO. > Up yetus proclimit on nightlies > --- > > Key: HBASE-19901 > URL: https://issues.apache.org/jira/browse/HBASE-19901 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-19901.master.001.patch, > HBASE-19901.master.002.patch > > > We're on 0.7.0 now which enforces limits meant to protect against runaway > processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. > Up our proclimit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies
[ https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349746#comment-16349746 ] stack commented on HBASE-19901: --- Arghh... After revert, nightlies got further... running unit tests... but failing in hbase-procedure... not saying why. > Up yetus proclimit on nightlies > --- > > Key: HBASE-19901 > URL: https://issues.apache.org/jira/browse/HBASE-19901 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-19901.master.001.patch, > HBASE-19901.master.002.patch > > > We're on 0.7.0 now which enforces limits meant to protect against runaway > processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. > Up our proclimit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies
[ https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349740#comment-16349740 ] Hudson commented on HBASE-19901: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4511 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4511/]) HBASE-19901 Up yetus proclimit on nightlies; ADDENDUM -- up proclimit (stack: rev 0db7db3cd1022f794c1cf6d83858e7c83964cd73) * (edit) dev-support/hbase_nightly_yetus.sh * (edit) dev-support/hbase-personality.sh HBASE-19901 Up yetus proclimit on nightlies; ADDENDUM -- remove docker (stack: rev 5f5ddf55416a09c9675a0026b75dfc2aaf12632d) * (edit) dev-support/hbase-personality.sh * (edit) dev-support/hbase_nightly_yetus.sh > Up yetus proclimit on nightlies > --- > > Key: HBASE-19901 > URL: https://issues.apache.org/jira/browse/HBASE-19901 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-19901.master.001.patch, > HBASE-19901.master.002.patch > > > We're on 0.7.0 now which enforces limits meant to protect against runaway > processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. > Up our proclimit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies
[ https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349667#comment-16349667 ] stack commented on HBASE-19901: --- Ok. Reverted original patch too. Back to square one. I can't get nightlies to build. Failing cryptically. Starting over. > Up yetus proclimit on nightlies > --- > > Key: HBASE-19901 > URL: https://issues.apache.org/jira/browse/HBASE-19901 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-19901.master.001.patch, > HBASE-19901.master.002.patch > > > We're on 0.7.0 now which enforces limits meant to protect against runaway > processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. > Up our proclimit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies
[ https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349597#comment-16349597 ] stack commented on HBASE-19901: --- Pushed a new addendum that removes docker limit for the moment. Having trouble getting nightlies to launch. This is what I pushed as addendum number two. commit 5f5ddf55416a09c9675a0026b75dfc2aaf12632d Author: Michael StackDate: Thu Feb 1 16:38:50 2018 -0800 HBASE-19901 Up yetus proclimit on nightlies; ADDENDUM -- remove docker mem sizing for the moment... diff --git a/dev-support/hbase-personality.sh b/dev-support/hbase-personality.sh index 5526e09d59..ddd0df4d2b 100755 --- a/dev-support/hbase-personality.sh +++ b/dev-support/hbase-personality.sh @@ -71,7 +71,7 @@ function personality_globals # Set docker container to run with 20g. Default is 4g in yetus. # See HBASE-19902 for how we arrived at 20g. - DOCKERMEMLIMIT=20g + # UNUSED AT MOMENT DOCKERMEMLIMIT=20g } ## @description Parse extra arguments required by personalities, if any. diff --git a/dev-support/hbase_nightly_yetus.sh b/dev-support/hbase_nightly_yetus.sh index 651a2e22d7..3e6fc7c97d 100755 --- a/dev-support/hbase_nightly_yetus.sh +++ b/dev-support/hbase_nightly_yetus.sh @@ -68,7 +68,6 @@ YETUS_ARGS=("--sentinel" "${YETUS_ARGS[@]}") YETUS_ARGS=("--branch=${BRANCH_NAME}" "${YETUS_ARGS[@]}") YETUS_ARGS=("--tests-filter=${TESTS_FILTER}" "${YETUS_ARGS[@]}") YETUS_ARGS=("--proclimit=${PROCLIMIT}" "${YETUS_ARGS[@]}") -YETUS_ARGS=("--dockermemlimit=${DOCKERMEMLIMIT}" "${YETUS_ARGS[@]}") # Currently, flaky list is calculated only for master branch. UNDERSCORED_BRANCH_NAME=$(echo ${BRANCH_NAME} | tr '.-' '_') > Up yetus proclimit on nightlies > --- > > Key: HBASE-19901 > URL: https://issues.apache.org/jira/browse/HBASE-19901 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-19901.master.001.patch, > HBASE-19901.master.002.patch > > > We're on 0.7.0 now which enforces limits meant to protect against runaway > processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. > Up our proclimit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies
[ https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349362#comment-16349362 ] Hadoop QA commented on HBASE-19901: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 3s{color} | {color:red} HBASE-19901 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HBASE-19901 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12908875/HBASE-19901.master.002.patch | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/11347/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > Up yetus proclimit on nightlies > --- > > Key: HBASE-19901 > URL: https://issues.apache.org/jira/browse/HBASE-19901 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-19901.master.001.patch, > HBASE-19901.master.002.patch > > > We're on 0.7.0 now which enforces limits meant to protect against runaway > processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. > Up our proclimit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies
[ https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349356#comment-16349356 ] stack commented on HBASE-19901: --- .002 is an addendum applied to branch-2 and master. It ups proclimit to 10k for now until we figure out why so many procs during our test runs... and it adds a docker containter 20g limit which ups default of 4g from yetus. > Up yetus proclimit on nightlies > --- > > Key: HBASE-19901 > URL: https://issues.apache.org/jira/browse/HBASE-19901 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-19901.master.001.patch, > HBASE-19901.master.002.patch > > > We're on 0.7.0 now which enforces limits meant to protect against runaway > processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. > Up our proclimit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies
[ https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348067#comment-16348067 ] stack commented on HBASE-19901: --- TODO: proclimit is still being worked on. memory for docker too. See tail of HBASE-19902 > Up yetus proclimit on nightlies > --- > > Key: HBASE-19901 > URL: https://issues.apache.org/jira/browse/HBASE-19901 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-19901.master.001.patch > > > We're on 0.7.0 now which enforces limits meant to protect against runaway > processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. > Up our proclimit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies
[ https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346733#comment-16346733 ] Hudson commented on HBASE-19901: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4501 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4501/]) HBASE-19901 HBASE-19901 Up yetus proclimit on nightlies (stack: rev 2e6bc12441f50a4de9ee87e96132cb067556a528) * (edit) dev-support/hbase_nightly_yetus.sh * (edit) dev-support/hbase-personality.sh > Up yetus proclimit on nightlies > --- > > Key: HBASE-19901 > URL: https://issues.apache.org/jira/browse/HBASE-19901 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-19901.master.001.patch > > > We're on 0.7.0 now which enforces limits meant to protect against runaway > processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. > Up our proclimit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies
[ https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346324#comment-16346324 ] stack commented on HBASE-19901: --- Pushed it. The shellcheck is good but I think this is the way hbase-personality is supposed to work making declarations. Will see. Pushed to master, branch-2, and branch-1. Leaving open to see if nightly picks up yetus change. > Up yetus proclimit on nightlies > --- > > Key: HBASE-19901 > URL: https://issues.apache.org/jira/browse/HBASE-19901 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-19901.master.001.patch > > > We're on 0.7.0 now which enforces limits meant to protect against runaway > processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. > Up our proclimit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies
[ https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346295#comment-16346295 ] Hadoop QA commented on HBASE-19901: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 3s{color} | {color:blue} Shelldocs was not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || || || || || {color:brown} Patch Compile Tests {color} || | {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m 1s{color} | {color:red} The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 11s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 0m 33s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 | | JIRA Issue | HBASE-19901 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12908502/HBASE-19901.master.001.patch | | Optional Tests | asflicense shellcheck shelldocs | | uname | Linux e4528ebffc35 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh | | git revision | master / 08bcd64c25 | | maven | version: Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) | | shellcheck | v0.4.4 | | shellcheck | https://builds.apache.org/job/PreCommit-HBASE-Build/11278/artifact/patchprocess/diff-patch-shellcheck.txt | | modules | C: . U: . | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/11278/console | | Powered by | Apache Yetus 0.6.0 http://yetus.apache.org | This message was automatically generated. > Up yetus proclimit on nightlies > --- > > Key: HBASE-19901 > URL: https://issues.apache.org/jira/browse/HBASE-19901 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-19901.master.001.patch > > > We're on 0.7.0 now which enforces limits meant to protect against runaway > processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. > Up our proclimit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies
[ https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346294#comment-16346294 ] Hadoop QA commented on HBASE-19901: --- (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HBASE-Build/11278/console in case of problems. > Up yetus proclimit on nightlies > --- > > Key: HBASE-19901 > URL: https://issues.apache.org/jira/browse/HBASE-19901 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-19901.master.001.patch > > > We're on 0.7.0 now which enforces limits meant to protect against runaway > processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. > Up our proclimit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)