[jira] [Commented] (YARN-7083) Log aggregation deletes/renames while file is open
[ https://issues.apache.org/jira/browse/YARN-7083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146054#comment-16146054 ] Junping Du commented on YARN-7083: -- Ok. I have filed YARN-7124 to fix for trunk and branch-2. Commit the patch to branch-2.8 and branch-2.8.2. Thanks [~daryn] for reporting the issue and [~jlowe] to deliver a fix! > Log aggregation deletes/renames while file is open > -- > > Key: YARN-7083 > URL: https://issues.apache.org/jira/browse/YARN-7083 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.8.2 >Reporter: Daryn Sharp >Assignee: Jason Lowe >Priority: Critical > Fix For: 2.8.2 > > Attachments: YARN-7083.001.patch > > > YARN-6288 changes the log aggregation writer to be an autoclosable. > Unfortunately the try-with-resources block for the writer will either rename > or delete the log while open. > Assuming the NM's behavior is correct, deleting open files only results in > ominous WARNs in the nodemanager log and increases the rate of logging in the > NN when the implicit try-with-resource close fails. These red herrings > complicate debugging efforts. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7083) Log aggregation deletes/renames while file is open
[ https://issues.apache.org/jira/browse/YARN-7083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144420#comment-16144420 ] Jason Lowe commented on YARN-7083: -- I'm OK with fixing in 2.8.x and filing a followup JIRA. If that followup isn't going to be fixed for a bit it may make more sense to revert YARN-6876 until it's ready to address the issue to avoid shipping this bug in 3.0.0-beta1 or 2.9.0. > Log aggregation deletes/renames while file is open > -- > > Key: YARN-7083 > URL: https://issues.apache.org/jira/browse/YARN-7083 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.8.2 >Reporter: Daryn Sharp >Assignee: Jason Lowe >Priority: Critical > Attachments: YARN-7083.001.patch > > > YARN-6288 changes the log aggregation writer to be an autoclosable. > Unfortunately the try-with-resources block for the writer will either rename > or delete the log while open. > Assuming the NM's behavior is correct, deleting open files only results in > ominous WARNs in the nodemanager log and increases the rate of logging in the > NN when the implicit try-with-resource close fails. These red herrings > complicate debugging efforts. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7083) Log aggregation deletes/renames while file is open
[ https://issues.apache.org/jira/browse/YARN-7083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144317#comment-16144317 ] Junping Du commented on YARN-7083: -- Trunk code has changed significantly since YARN-6877 so patch here doesn't apply any more. From my quick look, the issue (not close file before rename/delete) is still there but the fix is not straightforward as writer get hidden behind for different formats. I would suggest to commit the patch fix here to branch-2.8 and branch-2.8.2 only and create a separated jira tracking for trunk/branch-2. [~jlowe], what do you think? > Log aggregation deletes/renames while file is open > -- > > Key: YARN-7083 > URL: https://issues.apache.org/jira/browse/YARN-7083 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.8.2 >Reporter: Daryn Sharp >Assignee: Jason Lowe >Priority: Critical > Attachments: YARN-7083.001.patch > > > YARN-6288 changes the log aggregation writer to be an autoclosable. > Unfortunately the try-with-resources block for the writer will either rename > or delete the log while open. > Assuming the NM's behavior is correct, deleting open files only results in > ominous WARNs in the nodemanager log and increases the rate of logging in the > NN when the implicit try-with-resource close fails. These red herrings > complicate debugging efforts. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7083) Log aggregation deletes/renames while file is open
[ https://issues.apache.org/jira/browse/YARN-7083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143247#comment-16143247 ] Junping Du commented on YARN-7083: -- Thanks [~daryn] for reporting the issue and [~jlowe] for delivering a fix. The issue reported here is we were losing writer.close() in proper places since YARN-6288 as we were fully lean on auto close in the end. The fix here seems straightforward enough so I think no UT here should be fine. +1. I will commit it tomorrow if no further comments. > Log aggregation deletes/renames while file is open > -- > > Key: YARN-7083 > URL: https://issues.apache.org/jira/browse/YARN-7083 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.8.2 >Reporter: Daryn Sharp >Assignee: Jason Lowe >Priority: Critical > Attachments: YARN-7083.001.patch > > > YARN-6288 changes the log aggregation writer to be an autoclosable. > Unfortunately the try-with-resources block for the writer will either rename > or delete the log while open. > Assuming the NM's behavior is correct, deleting open files only results in > ominous WARNs in the nodemanager log and increases the rate of logging in the > NN when the implicit try-with-resource close fails. These red herrings > complicate debugging efforts. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7083) Log aggregation deletes/renames while file is open
[
https://issues.apache.org/jira/browse/YARN-7083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139127#comment-16139127
]
Hadoop QA commented on YARN-7083:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
40s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 15s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
40s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-7083 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12883413/YARN-7083.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d7a65b2fdf7e 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7e6463d |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/17097/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/17097/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/17097/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
|
