[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349984#comment-16349984
 ] 

Hudson commented on HBASE-19901:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4512 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/4512/])
Revert "HBASE-19901 HBASE-19901 Up yetus proclimit on nightlies" (stack: rev 
18eec8c1a58574ae3ff529904f677288ff696765)
* (edit) dev-support/hbase_nightly_yetus.sh
* (edit) dev-support/hbase-personality.sh
HBASE-19901 HBASE-19901 Up yetus proclimit on nightlies; REAPPLY TO TEST 
(stack: rev cb7bfc21da4e2658602a9a2b2080c4d883287dd1)
* (edit) dev-support/hbase_nightly_yetus.sh
* (edit) dev-support/hbase-personality.sh
HBASE-19901 HBASE-19901 Up yetus proclimit on nightlies; AMENDMENT (stack: rev 
a2bc19aa112b8cd0697845d0825d277ff3a8bcfc)
* (edit) dev-support/hbase_nightly_yetus.sh


> Up yetus proclimit on nightlies
> ---
>
> Key: HBASE-19901
> URL: https://issues.apache.org/jira/browse/HBASE-19901
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-19901.master.001.patch, 
> HBASE-19901.master.002.patch
>
>
> We're on 0.7.0 now which enforces limits meant to protect against runaway 
> processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. 
> Up our proclimit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349857#comment-16349857
 ] 

stack commented on HBASE-19901:
---

I reverted all changes made against this issue, then put back the originals. 
With DEBUG I see that the hbase_nightly is not picking up defines in 
hbase-personality... So nightly was failing because docker mem set to ... ? 
Ditto on proclimit. So hardcoded values in hbase_nightly. Not nice.  Nightly 
seems to be working now though. To be revisited. Here is what is committed 
currently...

commit b2fe241089dddaf65253ad882fbbff36225dafd2
Author: Michael Stack 
Date:   Thu Feb 1 20:54:13 2018 -0800

HBASE-19901 HBASE-19901 Up yetus proclimit on nightlies; AMENDMENT hardcode 
proclimit and docker memlimit in nightly script...

diff --git a/dev-support/hbase_nightly_yetus.sh 
b/dev-support/hbase_nightly_yetus.sh
index 651a2e22d7..27d11c8beb 100755
--- a/dev-support/hbase_nightly_yetus.sh
+++ b/dev-support/hbase_nightly_yetus.sh
@@ -67,8 +67,9 @@ 
YETUS_ARGS=("--whitespace-tabs-ignore-list=${WHITESPACE_IGNORE_LIST}" "${YETUS_A
 YETUS_ARGS=("--sentinel" "${YETUS_ARGS[@]}")
 YETUS_ARGS=("--branch=${BRANCH_NAME}" "${YETUS_ARGS[@]}")
 YETUS_ARGS=("--tests-filter=${TESTS_FILTER}" "${YETUS_ARGS[@]}")
-YETUS_ARGS=("--proclimit=${PROCLIMIT}" "${YETUS_ARGS[@]}")
-YETUS_ARGS=("--dockermemlimit=${DOCKERMEMLIMIT}" "${YETUS_ARGS[@]}")
+# Why are these not being picked up from hbase-personality?
+YETUS_ARGS=("--proclimit=1" "${YETUS_ARGS[@]}")
+YETUS_ARGS=("--dockermemlimit=20g" "${YETUS_ARGS[@]}")

 # Currently, flaky list is calculated only for master branch.
 UNDERSCORED_BRANCH_NAME=$(echo ${BRANCH_NAME} | tr '.-' '_')


The defines in hbase-personality:



  # Yetus 0.7.0 enforces limits. Default proclimit is 1000.
  # Up it. See HBASE-19902 for how we arrived at this number.
  PROCLIMIT=1

  # Set docker container to run with 20g. Default is 4g in yetus.
  # See HBASE-19902 for how we arrived at 20g.
  DOCKERMEMLIMIT=20g


... are what are not being picked up. I traced download of hbase-personality as 
personality.sh... but don't see how it picked up (though yetus describes 
picking up these globals...) TODO.

> Up yetus proclimit on nightlies
> ---
>
> Key: HBASE-19901
> URL: https://issues.apache.org/jira/browse/HBASE-19901
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-19901.master.001.patch, 
> HBASE-19901.master.002.patch
>
>
> We're on 0.7.0 now which enforces limits meant to protect against runaway 
> processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. 
> Up our proclimit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349746#comment-16349746
 ] 

stack commented on HBASE-19901:
---

Arghh... After revert, nightlies got further... running unit tests... but 
failing in hbase-procedure... not saying why.

> Up yetus proclimit on nightlies
> ---
>
> Key: HBASE-19901
> URL: https://issues.apache.org/jira/browse/HBASE-19901
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-19901.master.001.patch, 
> HBASE-19901.master.002.patch
>
>
> We're on 0.7.0 now which enforces limits meant to protect against runaway 
> processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. 
> Up our proclimit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349740#comment-16349740
 ] 

Hudson commented on HBASE-19901:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4511 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/4511/])
HBASE-19901 Up yetus proclimit on nightlies; ADDENDUM -- up proclimit (stack: 
rev 0db7db3cd1022f794c1cf6d83858e7c83964cd73)
* (edit) dev-support/hbase_nightly_yetus.sh
* (edit) dev-support/hbase-personality.sh
HBASE-19901 Up yetus proclimit on nightlies; ADDENDUM -- remove docker (stack: 
rev 5f5ddf55416a09c9675a0026b75dfc2aaf12632d)
* (edit) dev-support/hbase-personality.sh
* (edit) dev-support/hbase_nightly_yetus.sh


> Up yetus proclimit on nightlies
> ---
>
> Key: HBASE-19901
> URL: https://issues.apache.org/jira/browse/HBASE-19901
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-19901.master.001.patch, 
> HBASE-19901.master.002.patch
>
>
> We're on 0.7.0 now which enforces limits meant to protect against runaway 
> processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. 
> Up our proclimit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349667#comment-16349667
 ] 

stack commented on HBASE-19901:
---

Ok. Reverted original patch too. Back to square one. I can't get nightlies to 
build. Failing cryptically. Starting over.

> Up yetus proclimit on nightlies
> ---
>
> Key: HBASE-19901
> URL: https://issues.apache.org/jira/browse/HBASE-19901
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-19901.master.001.patch, 
> HBASE-19901.master.002.patch
>
>
> We're on 0.7.0 now which enforces limits meant to protect against runaway 
> processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. 
> Up our proclimit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349597#comment-16349597
 ] 

stack commented on HBASE-19901:
---

Pushed a new addendum that removes docker limit for the moment. Having trouble 
getting nightlies to launch. This is what I pushed as addendum number two.

commit 5f5ddf55416a09c9675a0026b75dfc2aaf12632d
Author: Michael Stack 
Date:   Thu Feb 1 16:38:50 2018 -0800

HBASE-19901 Up yetus proclimit on nightlies; ADDENDUM -- remove docker mem 
sizing for the moment...

diff --git a/dev-support/hbase-personality.sh b/dev-support/hbase-personality.sh
index 5526e09d59..ddd0df4d2b 100755
--- a/dev-support/hbase-personality.sh
+++ b/dev-support/hbase-personality.sh
@@ -71,7 +71,7 @@ function personality_globals

   # Set docker container to run with 20g. Default is 4g in yetus.
   # See HBASE-19902 for how we arrived at 20g.
-  DOCKERMEMLIMIT=20g
+  # UNUSED AT MOMENT DOCKERMEMLIMIT=20g
 }

 ## @description  Parse extra arguments required by personalities, if any.
diff --git a/dev-support/hbase_nightly_yetus.sh 
b/dev-support/hbase_nightly_yetus.sh
index 651a2e22d7..3e6fc7c97d 100755
--- a/dev-support/hbase_nightly_yetus.sh
+++ b/dev-support/hbase_nightly_yetus.sh
@@ -68,7 +68,6 @@ YETUS_ARGS=("--sentinel" "${YETUS_ARGS[@]}")
 YETUS_ARGS=("--branch=${BRANCH_NAME}" "${YETUS_ARGS[@]}")
 YETUS_ARGS=("--tests-filter=${TESTS_FILTER}" "${YETUS_ARGS[@]}")
 YETUS_ARGS=("--proclimit=${PROCLIMIT}" "${YETUS_ARGS[@]}")
-YETUS_ARGS=("--dockermemlimit=${DOCKERMEMLIMIT}" "${YETUS_ARGS[@]}")

 # Currently, flaky list is calculated only for master branch.
 UNDERSCORED_BRANCH_NAME=$(echo ${BRANCH_NAME} | tr '.-' '_')

> Up yetus proclimit on nightlies
> ---
>
> Key: HBASE-19901
> URL: https://issues.apache.org/jira/browse/HBASE-19901
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-19901.master.001.patch, 
> HBASE-19901.master.002.patch
>
>
> We're on 0.7.0 now which enforces limits meant to protect against runaway 
> processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. 
> Up our proclimit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349362#comment-16349362
 ] 

Hadoop QA commented on HBASE-19901:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  3s{color} 
| {color:red} HBASE-19901 does not apply to master. Rebase required? Wrong 
Branch? See https://yetus.apache.org/documentation/0.7.0/precommit-patchnames 
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-19901 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12908875/HBASE-19901.master.002.patch
 |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11347/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Up yetus proclimit on nightlies
> ---
>
> Key: HBASE-19901
> URL: https://issues.apache.org/jira/browse/HBASE-19901
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-19901.master.001.patch, 
> HBASE-19901.master.002.patch
>
>
> We're on 0.7.0 now which enforces limits meant to protect against runaway 
> processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. 
> Up our proclimit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349356#comment-16349356
 ] 

stack commented on HBASE-19901:
---

.002 is an addendum applied to branch-2 and master.

It ups proclimit to 10k for now until we figure out why so many procs during 
our test runs... and it adds a docker containter 20g limit which ups default of 
4g from yetus.

> Up yetus proclimit on nightlies
> ---
>
> Key: HBASE-19901
> URL: https://issues.apache.org/jira/browse/HBASE-19901
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-19901.master.001.patch, 
> HBASE-19901.master.002.patch
>
>
> We're on 0.7.0 now which enforces limits meant to protect against runaway 
> processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. 
> Up our proclimit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348067#comment-16348067
 ] 

stack commented on HBASE-19901:
---

TODO: proclimit is still being worked on. memory for docker too. See tail of 
HBASE-19902

> Up yetus proclimit on nightlies
> ---
>
> Key: HBASE-19901
> URL: https://issues.apache.org/jira/browse/HBASE-19901
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-19901.master.001.patch
>
>
> We're on 0.7.0 now which enforces limits meant to protect against runaway 
> processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. 
> Up our proclimit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346733#comment-16346733
 ] 

Hudson commented on HBASE-19901:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4501 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/4501/])
HBASE-19901 HBASE-19901 Up yetus proclimit on nightlies (stack: rev 
2e6bc12441f50a4de9ee87e96132cb067556a528)
* (edit) dev-support/hbase_nightly_yetus.sh
* (edit) dev-support/hbase-personality.sh


> Up yetus proclimit on nightlies
> ---
>
> Key: HBASE-19901
> URL: https://issues.apache.org/jira/browse/HBASE-19901
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-19901.master.001.patch
>
>
> We're on 0.7.0 now which enforces limits meant to protect against runaway 
> processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. 
> Up our proclimit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346324#comment-16346324
 ] 

stack commented on HBASE-19901:
---

Pushed it. The shellcheck is good but I think this is the way hbase-personality 
is supposed to work making declarations. Will see. Pushed to master, branch-2, 
and branch-1. Leaving open to see if nightly picks up yetus change.

> Up yetus proclimit on nightlies
> ---
>
> Key: HBASE-19901
> URL: https://issues.apache.org/jira/browse/HBASE-19901
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-19901.master.001.patch
>
>
> We're on 0.7.0 now which enforces limits meant to protect against runaway 
> processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. 
> Up our proclimit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346295#comment-16346295
 ] 

Hadoop QA commented on HBASE-19901:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue}  0m  
3s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red}  0m  
1s{color} | {color:red} The patch generated 1 new + 1 unchanged - 0 fixed = 2 
total (was 1) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
11s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}  0m 33s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19901 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12908502/HBASE-19901.master.001.patch
 |
| Optional Tests |  asflicense  shellcheck  shelldocs  |
| uname | Linux e4528ebffc35 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / 08bcd64c25 |
| maven | version: Apache Maven 3.5.2 
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| shellcheck | v0.4.4 |
| shellcheck | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11278/artifact/patchprocess/diff-patch-shellcheck.txt
 |
| modules | C: . U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/11278/console |
| Powered by | Apache Yetus 0.6.0   http://yetus.apache.org |


This message was automatically generated.



> Up yetus proclimit on nightlies
> ---
>
> Key: HBASE-19901
> URL: https://issues.apache.org/jira/browse/HBASE-19901
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-19901.master.001.patch
>
>
> We're on 0.7.0 now which enforces limits meant to protect against runaway 
> processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. 
> Up our proclimit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-19901) Up yetus proclimit on nightlies

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-19901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346294#comment-16346294
 ] 

Hadoop QA commented on HBASE-19901:
---

(!) A patch to the testing environment has been detected. 
Re-executing against the patched versions to perform further tests. 
The console is at 
https://builds.apache.org/job/PreCommit-HBASE-Build/11278/console in case of 
problems.


> Up yetus proclimit on nightlies
> ---
>
> Key: HBASE-19901
> URL: https://issues.apache.org/jira/browse/HBASE-19901
> Project: HBase
>  Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: HBASE-19901.master.001.patch
>
>
> We're on 0.7.0 now which enforces limits meant to protect against runaway 
> processes. Default is 1000 procs. HBase test runs seem to consume almost 4k. 
> Up our proclimit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)