[jira] [Commented] (HADOOP-15180) branch-2 : daemon processes' sysout overwrites 'ulimit -a' in daemon's out file
[
https://issues.apache.org/jira/browse/HADOOP-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442890#comment-16442890
]
Brahma Reddy Battula commented on HADOOP-15180:
---
+1 on latest patch.will commit soon
> branch-2 : daemon processes' sysout overwrites 'ulimit -a' in daemon's out
> file
> ---
>
> Key: HADOOP-15180
> URL: https://issues.apache.org/jira/browse/HADOOP-15180
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 2.7.2
>Reporter: Ranith Sardar
>Assignee: Ranith Sardar
>Priority: Minor
> Attachments: HADOOP-15180-branch-2-002.patch,
> HADOOP-15180_branch-2.diff
>
>
> Whenever the balancer starts, it will redirect the sys out to the out log
> file. And balancer writes the system output to the log file, at the same
> time script also will try to append ulimit output.
> {noformat}
> # capture the ulimit output
> if [ "true" = "$starting_secure_dn" ]; then
> echo "ulimit -a for secure datanode user $HADOOP_SECURE_DN_USER" >> $log
> # capture the ulimit info for the appropriate user
> su --shell=/bin/bash $HADOOP_SECURE_DN_USER -c 'ulimit -a' >> $log 2>&1
> elif [ "true" = "$starting_privileged_nfs" ]; then
> echo "ulimit -a for privileged nfs user $HADOOP_PRIVILEGED_NFS_USER"
> >> $log
> su --shell=/bin/bash $HADOOP_PRIVILEGED_NFS_USER -c 'ulimit -a' >>
> $log 2>&1
> else
> echo "ulimit -a for user $USER" >> $log
> ulimit -a >> $log 2>&1
> fi
> sleep 3;
> if ! ps -p $! > /dev/null ; then
> exit 1
> fi
> {noformat}
> But the problem is first few lines of ulimit is overridding by the log of
> balancer.
> {noformat}
> vm1:/opt/install/hadoop/namenode/sbin # cat
> /opt/HA/AIH283/install/hadoop/namenode/logs/hadoop-root-balancer-vm1.out
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
> Bytes Being Moved
> The cluster is balanced. Exiting...
> Jan 9, 2018 6:26:26 PM0 0 B 0 B
> 0 B
> Jan 9, 2018 6:26:26 PM Balancing took 3.446 seconds
> x memory size (kbytes, -m) 13428300
> open files (-n) 1024
> pipe size(512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 127350
> virtual memory (kbytes, -v) 15992160
> file locks (-x) unlimited
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15180) branch-2 : daemon processes' sysout overwrites 'ulimit -a' in daemon's out file
[
https://issues.apache.org/jira/browse/HADOOP-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437515#comment-16437515
]
genericqa commented on HADOOP-15180:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 26m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 9m
48s{color} | {color:red} root in branch-2 failed. {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
10s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
51s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:f667ef1 |
| JIRA Issue | HADOOP-15180 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918967/HADOOP-15180-branch-2-002.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux fc6177372a37 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / a772108 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| mvninstall |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14488/artifact/out/branch-mvninstall-root.txt
|
| shellcheck | v0.4.7 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14488/testReport/ |
| Max. process+thread count | 77 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14488/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> branch-2 : daemon processes' sysout overwrites 'ulimit -a' in daemon's out
> file
> ---
>
> Key: HADOOP-15180
> URL: https://issues.apache.org/jira/browse/HADOOP-15180
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 2.7.2
>Reporter: Ranith Sardar
>Assignee: Ranith Sardar
>Priority: Minor
> Attachments: HADOOP-15180-branch-2-002.patch,
> HADOOP-15180_branch-2.diff
>
>
> Whenever the balancer starts, it will redirect the sys out to the out log
> file. And balancer writes the system output to the log file, at the same
> time script also will try to append ulimit output.
> {noformat}
> # capture the ulimit output
> if [ "true" = "$starting_secure_dn" ]; then
> echo "ulimit -a for secure datanode user $HADOOP_SECURE_DN_USER" >> $log
> # capture the ulimit info for the appropriate user
> su --shell=/bin/bash $HADOOP_SECURE_DN_USER -c 'ulimit -a' >> $log 2>&1
> elif [ "true" = "$starting_privileged_nfs" ]; then
> echo "ulimit -a for privileged nfs user $HADOOP_PRIVILEGED_NFS_USER"
> >> $log
> su
[jira] [Commented] (HADOOP-15180) branch-2 : daemon processes' sysout overwrites 'ulimit -a' in daemon's out file
[
https://issues.apache.org/jira/browse/HADOOP-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437398#comment-16437398
]
Ranith Sardar commented on HADOOP-15180:
Thanks [~brahmareddy] for your review and Thanks [~vinayrpet] for assigning me.
I have updated the patch name. Please review it once.
> branch-2 : daemon processes' sysout overwrites 'ulimit -a' in daemon's out
> file
> ---
>
> Key: HADOOP-15180
> URL: https://issues.apache.org/jira/browse/HADOOP-15180
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 2.7.2
>Reporter: Ranith Sardar
>Assignee: Ranith Sardar
>Priority: Minor
> Attachments: HADOOP-15180_branch-2.diff
>
>
> Whenever the balancer starts, it will redirect the sys out to the out log
> file. And balancer writes the system output to the log file, at the same
> time script also will try to append ulimit output.
> {noformat}
> # capture the ulimit output
> if [ "true" = "$starting_secure_dn" ]; then
> echo "ulimit -a for secure datanode user $HADOOP_SECURE_DN_USER" >> $log
> # capture the ulimit info for the appropriate user
> su --shell=/bin/bash $HADOOP_SECURE_DN_USER -c 'ulimit -a' >> $log 2>&1
> elif [ "true" = "$starting_privileged_nfs" ]; then
> echo "ulimit -a for privileged nfs user $HADOOP_PRIVILEGED_NFS_USER"
> >> $log
> su --shell=/bin/bash $HADOOP_PRIVILEGED_NFS_USER -c 'ulimit -a' >>
> $log 2>&1
> else
> echo "ulimit -a for user $USER" >> $log
> ulimit -a >> $log 2>&1
> fi
> sleep 3;
> if ! ps -p $! > /dev/null ; then
> exit 1
> fi
> {noformat}
> But the problem is first few lines of ulimit is overridding by the log of
> balancer.
> {noformat}
> vm1:/opt/install/hadoop/namenode/sbin # cat
> /opt/HA/AIH283/install/hadoop/namenode/logs/hadoop-root-balancer-vm1.out
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
> Bytes Being Moved
> The cluster is balanced. Exiting...
> Jan 9, 2018 6:26:26 PM0 0 B 0 B
> 0 B
> Jan 9, 2018 6:26:26 PM Balancing took 3.446 seconds
> x memory size (kbytes, -m) 13428300
> open files (-n) 1024
> pipe size(512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 127350
> virtual memory (kbytes, -v) 15992160
> file locks (-x) unlimited
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15180) branch-2 : daemon processes' sysout overwrites 'ulimit -a' in daemon's out file
[
https://issues.apache.org/jira/browse/HADOOP-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437358#comment-16437358
]
Brahma Reddy Battula commented on HADOOP-15180:
---
[~RANith] thanks for reporting this issue.. Changes LGTM.
can you please update the patch to trigger the Jenkins ( upload patch like
HADOOP-15180-branch-2-**).
> branch-2 : daemon processes' sysout overwrites 'ulimit -a' in daemon's out
> file
> ---
>
> Key: HADOOP-15180
> URL: https://issues.apache.org/jira/browse/HADOOP-15180
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Ranith Sardar
>Assignee: Ranith Sardar
>Priority: Minor
> Attachments: HADOOP-15180_branch-2.diff
>
>
> Whenever the balancer starts, it will redirect the sys out to the out log
> file. And balancer writes the system output to the log file, at the same
> time script also will try to append ulimit output.
> {noformat}
> # capture the ulimit output
> if [ "true" = "$starting_secure_dn" ]; then
> echo "ulimit -a for secure datanode user $HADOOP_SECURE_DN_USER" >> $log
> # capture the ulimit info for the appropriate user
> su --shell=/bin/bash $HADOOP_SECURE_DN_USER -c 'ulimit -a' >> $log 2>&1
> elif [ "true" = "$starting_privileged_nfs" ]; then
> echo "ulimit -a for privileged nfs user $HADOOP_PRIVILEGED_NFS_USER"
> >> $log
> su --shell=/bin/bash $HADOOP_PRIVILEGED_NFS_USER -c 'ulimit -a' >>
> $log 2>&1
> else
> echo "ulimit -a for user $USER" >> $log
> ulimit -a >> $log 2>&1
> fi
> sleep 3;
> if ! ps -p $! > /dev/null ; then
> exit 1
> fi
> {noformat}
> But the problem is first few lines of ulimit is overridding by the log of
> balancer.
> {noformat}
> vm1:/opt/install/hadoop/namenode/sbin # cat
> /opt/HA/AIH283/install/hadoop/namenode/logs/hadoop-root-balancer-vm1.out
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
> Bytes Being Moved
> The cluster is balanced. Exiting...
> Jan 9, 2018 6:26:26 PM0 0 B 0 B
> 0 B
> Jan 9, 2018 6:26:26 PM Balancing took 3.446 seconds
> x memory size (kbytes, -m) 13428300
> open files (-n) 1024
> pipe size(512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 127350
> virtual memory (kbytes, -v) 15992160
> file locks (-x) unlimited
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15180) branch-2 : daemon processes' sysout overwrites 'ulimit -a' in daemon's out file
[
https://issues.apache.org/jira/browse/HADOOP-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16371384#comment-16371384
]
genericqa commented on HADOOP-15180:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
1s{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-15180 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15180 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911382/HADOOP-15180_branch-2.diff
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14179/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> branch-2 : daemon processes' sysout overwrites 'ulimit -a' in daemon's out
> file
> ---
>
> Key: HADOOP-15180
> URL: https://issues.apache.org/jira/browse/HADOOP-15180
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Ranith Sardar
>Assignee: Ranith Sardar
>Priority: Minor
> Attachments: HADOOP-15180_branch-2.diff
>
>
> Whenever the balancer starts, it will redirect the sys out to the out log
> file. And balancer writes the system output to the log file, at the same
> time script also will try to append ulimit output.
> {noformat}
> # capture the ulimit output
> if [ "true" = "$starting_secure_dn" ]; then
> echo "ulimit -a for secure datanode user $HADOOP_SECURE_DN_USER" >> $log
> # capture the ulimit info for the appropriate user
> su --shell=/bin/bash $HADOOP_SECURE_DN_USER -c 'ulimit -a' >> $log 2>&1
> elif [ "true" = "$starting_privileged_nfs" ]; then
> echo "ulimit -a for privileged nfs user $HADOOP_PRIVILEGED_NFS_USER"
> >> $log
> su --shell=/bin/bash $HADOOP_PRIVILEGED_NFS_USER -c 'ulimit -a' >>
> $log 2>&1
> else
> echo "ulimit -a for user $USER" >> $log
> ulimit -a >> $log 2>&1
> fi
> sleep 3;
> if ! ps -p $! > /dev/null ; then
> exit 1
> fi
> {noformat}
> But the problem is first few lines of ulimit is overridding by the log of
> balancer.
> {noformat}
> vm1:/opt/install/hadoop/namenode/sbin # cat
> /opt/HA/AIH283/install/hadoop/namenode/logs/hadoop-root-balancer-vm1.out
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
> Bytes Being Moved
> The cluster is balanced. Exiting...
> Jan 9, 2018 6:26:26 PM0 0 B 0 B
> 0 B
> Jan 9, 2018 6:26:26 PM Balancing took 3.446 seconds
> x memory size (kbytes, -m) 13428300
> open files (-n) 1024
> pipe size(512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 127350
> virtual memory (kbytes, -v) 15992160
> file locks (-x) unlimited
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
