[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212524#comment-17212524
]
Eric Badger commented on YARN-9667:
---
Thanks, [~Jim_Brennan]!
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.3.0, 3.2.2, 3.1.5, 2.10.2
>
> Attachments: YARN-9667-001.patch, YARN-9667-branch-2.10.001.patch,
> YARN-9667-branch-3.2.001.patch
>
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210539#comment-17210539
]
Hadoop QA commented on YARN-9667:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
22s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 1 new or modified
test files. {color} |
|| || || || {color:brown} branch-2.10 Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
29s{color} | | {color:green} branch-2.10 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | | {color:green} branch-2.10 passed with JDK Oracle
Corporation-1.7.0_95-b00 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | | {color:green} branch-2.10 passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | | {color:green} branch-2.10 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | | {color:green} the patch passed with JDK Oracle
Corporation-1.7.0_95-b00 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
52s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} golang {color} | {color:green} 0m
52s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
55s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} golang {color} | {color:green} 0m
55s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m
14s{color} | | {color:green} hadoop-yarn-server-nodemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 53s{color} |
| {color:black}{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/223/artifact/out/Dockerfile
|
| JIRA Issue | YARN-9667 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13013272/YARN-9667-branch-2.10.001.patch
|
| Optional Tests | dupname asflicense compile cc mvnsite javac unit golang |
| uname | Linux f8ef07144d98 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | branch-2.10 / deb35a32bafdd3065e3c2f243d84ef79209838e9 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-7-openjdk-amd64:Oracle
Corporation-1.7.0_95-b00 /usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/223/testReport/ |
| Max. process+thread count | 168 (vs. ulimit of 5500) |
| modules | C:
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210518#comment-17210518
]
Eric Badger commented on YARN-9667:
---
Thanks, [~Jim_Brennan]! I attached another patch that should work for
branch-2.10
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.3.0, 3.2.2, 3.1.5
>
> Attachments: YARN-9667-001.patch, YARN-9667-branch-2.10.001.patch,
> YARN-9667-branch-3.2.001.patch
>
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210465#comment-17210465
]
Jim Brennan commented on YARN-9667:
---
I've committed this to branch-3.2 and branch-3.1, but the patch does not apply
to branch-2.10.
[~ebadger] can you provide a patch for branch-2.10?
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.3.0, 3.2.2, 3.1.5
>
> Attachments: YARN-9667-001.patch, YARN-9667-branch-3.2.001.patch
>
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210457#comment-17210457
]
Jim Brennan commented on YARN-9667:
---
+1 on the patch for branch-3.2
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9667-001.patch, YARN-9667-branch-3.2.001.patch
>
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210351#comment-17210351
]
Hadoop QA commented on YARN-9667:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
21s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 1 new or modified
test files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
43s{color} | | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
36m 45s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
53s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} golang {color} | {color:green} 0m
53s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 39s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
6s{color} | | {color:green} hadoop-yarn-server-nodemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 4s{color} |
| {color:black}{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/221/artifact/out/Dockerfile
|
| JIRA Issue | YARN-9667 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13013218/YARN-9667-branch-3.2.001.patch
|
| Optional Tests | dupname asflicense compile cc mvnsite javac unit golang |
| uname | Linux 5f34f0053c74 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | branch-3.2 / f83e07a20f8de7f668f3a6dd2e871df71d93316b |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~16.04-b01 |
| Test Results |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/221/testReport/ |
| Max. process+thread count | 309 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/221/console |
| versions | git=2.7.4 maven=3.3.9 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
>
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901762#comment-16901762
]
Szilard Nemeth commented on YARN-9667:
--
Thanks [~eyang]!
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9667-001.patch
>
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900285#comment-16900285
]
Hudson commented on YARN-9667:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17042 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17042/])
YARN-9667. Use setbuf with line buffer to reduce fflush complexity in (eyang:
rev d6697da5e854355ac3718a85006b73315d0702aa)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.h
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.c
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/devices/devices-module.c
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9667-001.patch
>
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900278#comment-16900278
]
Eric Yang commented on YARN-9667:
-
[~pbacsko] Thanks for the explanation. _IOLBF takes care of the flush in
production code, hence no enhancement is required. +1 for patch 001.
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9667-001.patch
>
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900236#comment-16900236
]
Peter Bacsko commented on YARN-9667:
Ah sorry it wasn't clear. I kept those only because they occur before a call to
{{fork()}}. So basically whatever is inside the buffers, it will be forced out.
Ideally nothing should be there, but it's a good safety net.
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9667-001.patch
>
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900235#comment-16900235
]
Eric Yang commented on YARN-9667:
-
[~pbacsko] {quote}I kept the existing fprintf/fflush pair in the tests because
it happens before fork() and it feels like a good practice. In fact, we can add
this to production code as well.{quote}
I don't understand the statement here. It sounds like you prefer to keep
fprintf/fflush in production code? The patch is moving in opposite direction
to remove fflush. Can you elaborate which direction is preferred?
Thank you for the patch. I tested this patch on my cluster this morning and it
worked fine.
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9667-001.patch
>
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900227#comment-16900227
]
Peter Bacsko commented on YARN-9667:
[~eyang] [~snemeth] pls check this out when you have some time.
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9667-001.patch
>
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900226#comment-16900226
]
Peter Bacsko commented on YARN-9667:
Changes in the patch:
# Removed unnecessary \{{fflush()}} calls (mostly all of them)
# Added missing "\n" to the {{fprintf()}} calls that forces {{fflush()}} due
to {{_IOLBF}}
# Added {{setvbuf()}} to {{main.c}} and to the unit tests
I didn't modify the child process handling logic to call {{_exit()}}, let's
discuss that in a separate JIRA.
I kept the existing {{fprintf}}/{{fflush}} pair in the tests because it happens
before {{fork()}} and it feels like a good practice. In fact, we can add this
to production code as well.
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9667-001.patch
>
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900225#comment-16900225
]
Hadoop QA commented on YARN-9667:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
31m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
12s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | YARN-9667 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12976707/YARN-9667-001.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 1e7d18c37fb5 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 71aad60 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/24470/testReport/ |
| Max. process+thread count | 446 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24470/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-9667-001.patch
>
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
>
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898039#comment-16898039
]
Szilard Nemeth commented on YARN-9667:
--
Hi [~adam.antal]!
As discussed offline with [~pbacsko], we will create a separate jira to cover
the log message additions as it wants to achieve different things than this
jira.
Filed YARN-9717
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897190#comment-16897190
]
Szilard Nemeth commented on YARN-9667:
--
Hi [~adam.antal]!
The most important one is this:
There's a function called create_container_directories in container-executor.c.
You should place a log line like this:
{code:java}
fprintf(LOGFILE, "**Creating directory: %s, user: %s", container_dir, user);
{code}
Before we're calling:
{code:java}
if (mkdirs(container_dir, perms) == 0) {
result = 0;
}
{code}
If you want to be very precise, you can add some logging into the mkdirs
function as well, alternatively, before every call of mkdirs but I'd prefer the
former approach!
Thanks!
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897167#comment-16897167
]
Adam Antal commented on YARN-9667:
--
During some downstream testing we bumped into some problems with the container
executor where an extra logging would be quite helpful when local files and
directories could not be created (container-executor.c:1810).
If we plan to modify the fprintf/fflush pairs as [~pbacsko] suggested, could we
add this extra logging to the container-exeucutor?
[~snemeth] what exact error message you suggest?
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Peter Bacsko
>Priority: Major
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16889002#comment-16889002
]
Peter Bacsko commented on YARN-9667:
[~eyang] sure, I'm on vacation right now, but I'll upload a patch as soon as I
get back.
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Priority: Major
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888184#comment-16888184
]
Eric Yang commented on YARN-9667:
-
[~pbacsko] Thank you for the information. Would you like to generate a patch
for this?
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Priority: Major
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885318#comment-16885318
]
Adam Antal commented on YARN-9667:
--
The error message reproduced intermittently in our setup downstream, but I will
try to provide some more information soon.
The suggestions you mentioned are seemingly good to me, [~pbacsko].
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Priority: Major
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882463#comment-16882463
]
Peter Bacsko commented on YARN-9667:
[~eyang] I'm not exactly sure how to properly reproduce this, it was observed
by [~adam.antal] so he can provide additional details.
Regarding {{BUFSIZ}}, this seems to be a platform-dependent macro value.
Quoting from
[https://www.gnu.org/software/libc/manual/html_node/Controlling-Buffering.html]:
_"The value of this macro is an integer constant expression that is good to
use for the size argument to {{setvbuf}}. This value is guaranteed to be at
least {{256}}._
_The value of {{BUFSIZ}} is chosen on each system so as to make stream I/O
efficient. So it is a good idea to use {{BUFSIZ}} as the size for the buffer
when you call {{setvbuf}}."_
Looks like it's defined in {{stdio.h}}.
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Priority: Major
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882460#comment-16882460
]
Eric Yang commented on YARN-9667:
-
[~pbacsko] I am having difficulty reproducing this. I don't see the double
flush in my environment using trunk code base. Can you provide a procedure to
reproduce this?
_IOLBF is fine or _IOFBF with good BUFSIZ. What is the proposed BUFSIZ?
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Priority: Major
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881981#comment-16881981
]
Peter Bacsko commented on YARN-9667:
ping [~eyang]
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Priority: Major
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880582#comment-16880582
]
Peter Bacsko commented on YARN-9667:
I just found that this is probably even better: {{setvbuf(ERRORFILE, NULL,
_IOLBF, BUFSIZ); }}. This makes stderr line buffered, which is probably what we
want.
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Priority: Major
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880552#comment-16880552
]
Peter Bacsko commented on YARN-9667:
Oh, thanks for saying - I didn't see that it's not you.
Anyway, I recommended #1 because already have a bunch of {{fprintf}}/{{fflush}}
pair scattered around the code. So it's repetitive code that we should get rid
of. I don't know whether we flush at all {{fprintf}} but we do it quite often.
I was under the impression that it's deliberate - and if so, we might as well
disable stdio buffering completely. But it's something that we can discuss
online.
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Priority: Major
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880544#comment-16880544
]
Eric Yang commented on YARN-9667:
-
[~pbacsko] There is duplicated Eric Yang in jira that is a different person.
Please make sure that you use the correct user name. The proposed solution is
sounding. I think making flush before fork is good. We don't want to flush on
every log entry, otherwise, it reduces performance. Thanks
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Priority: Major
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880280#comment-16880280
]
Peter Bacsko commented on YARN-9667:
cc [~shaneku...@gmail.com] also
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Priority: Major
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880234#comment-16880234
]
Peter Bacsko commented on YARN-9667:
cc [~aceric] what do you think? Not sure who should be involved to discuss this.
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Priority: Major
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9667) Container-executor.c duplicates messages to stdout
[
https://issues.apache.org/jira/browse/YARN-9667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880230#comment-16880230
]
Peter Bacsko commented on YARN-9667:
I suggest considering two things:
1) When setting ERRORFILE or LOGFILE, just invoke {{setbuf(ERRORFILE, NULL);}}
and {{setbuf(LOGFILE, NULL);}}. Then writing the output will be written to the
stream immediately without buffering.
2) We should use {{_exit()}} in the child process. After doing some research,
this is the preferred method of exiting from a child process. There are some
exceptions, but in general this seems to be the commonly used way of
terminating a child. Eg. if {{execvp()}} fails then just bail out, avoid any
sort of cleanup. This avoids flusing buffers twice, plus avoids calling
functions registered with {{atexit()}} twice, which probably we don't have
right now, but guarding against a potential bug is always a good thing.
> Container-executor.c duplicates messages to stdout
> --
>
> Key: YARN-9667
> URL: https://issues.apache.org/jira/browse/YARN-9667
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Priority: Major
>
> When a container is killed by its AM we get a similar error message like this:
> {noformat}
> 2019-06-30 12:09:04,412 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor:
> Shell execution returned exit code: 143. Privileged Execution Operation
> Stderr:
> Stdout: main : command provided 1
> main : run as user is systest
> main : requested yarn user is systest
> Getting exit code file...
> Creating script paths...
> Writing pid file...
> Writing to tmp file
> /yarn/nm/nmPrivate/application_1561921629886_0001/container_e84_1561921629886_0001_01_19/container_e84_1561921629886_0001_01_19.pid.tmp
> Writing to cgroup task files...
> Creating local dirs...
> Launching container...
> Getting exit code file...
> Creating script paths...
> {noformat}
> In container-executor.c the fork point is right after the "Creating script
> paths..." part, though in the Stdout log we can clearly see it has been
> written there twice. After consulting with [~pbacsko] it seems like there's a
> missing flush in container-executor.c before the fork and that causes the
> duplication.
> I suggest to add a flush there so that it won't be duplicated: it's a bit
> misleading that the child process writes out "Getting exit code file" and
> "Creating script paths" even though it is clearly not doing that.
> A more appealing solution could be to revisit the fprintf-fflush pairs in the
> code and change them to a single call, so that the fflush calls would not be
> forgotten accidentally. (It can cause problems in every place where it's
> used).
> Note: this issue probably affects every occasion of fork(), not just the one
> from {{launch_container_as_user}} in {{main.c}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
