[jira] [Commented] (HADOOP-14902) LoadGenerator#genFile write close timing is incorrectly calculated
[
https://issues.apache.org/jira/browse/HADOOP-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451039#comment-16451039
]
Hudson commented on HADOOP-14902:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14057 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14057/])
HADOOP-14902. LoadGenerator#genFile write close timing is incorrectly (xyao:
rev 16c6299088edfffe6d0a1c5d72498551256cfe86)
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/loadGenerator/LoadGenerator.java
> LoadGenerator#genFile write close timing is incorrectly calculated
> --
>
> Key: HADOOP-14902
> URL: https://issues.apache.org/jira/browse/HADOOP-14902
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.4.0
>Reporter: Jason Lowe
>Assignee: Hanisha Koneru
>Priority: Major
> Fix For: 2.9.0, 2.8.3, 2.7.5, 3.0.0
>
> Attachments: HADOOP-14902.001.patch, HADOOP-14902.002.patch,
> HADOOP-14902.003.patch
>
>
> LoadGenerator#genFile's write close timing code looks like the following:
> {code}
> startTime = Time.now();
> executionTime[WRITE_CLOSE] += (Time.now() - startTime);
> {code}
> That code will generate a zero (or near zero) write close timing since it
> isn't actually closing the file in-between timestamp lookups.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14902) LoadGenerator#genFile write close timing is incorrectly calculated
[
https://issues.apache.org/jira/browse/HADOOP-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16185065#comment-16185065
]
Hudson commented on HADOOP-14902:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12993 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12993/])
HADOOP-14902. LoadGenerator#genFile write close timing is incorrectly (jlowe:
rev 6f789fe05766a61b12ca10df3f26ee354eac84aa)
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/loadGenerator/LoadGenerator.java
> LoadGenerator#genFile write close timing is incorrectly calculated
> --
>
> Key: HADOOP-14902
> URL: https://issues.apache.org/jira/browse/HADOOP-14902
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.4.0
>Reporter: Jason Lowe
>Assignee: Hanisha Koneru
> Fix For: 2.9.0, 2.8.3, 2.7.5, 3.0.0
>
> Attachments: HADOOP-14902.001.patch, HADOOP-14902.002.patch,
> HADOOP-14902.003.patch
>
>
> LoadGenerator#genFile's write close timing code looks like the following:
> {code}
> startTime = Time.now();
> executionTime[WRITE_CLOSE] += (Time.now() - startTime);
> {code}
> That code will generate a zero (or near zero) write close timing since it
> isn't actually closing the file in-between timestamp lookups.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14902) LoadGenerator#genFile write close timing is incorrectly calculated
[
https://issues.apache.org/jira/browse/HADOOP-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184997#comment-16184997
]
Hanisha Koneru commented on HADOOP-14902:
-
Thanks [~jlowe] for reviewing and committing the patch.
> LoadGenerator#genFile write close timing is incorrectly calculated
> --
>
> Key: HADOOP-14902
> URL: https://issues.apache.org/jira/browse/HADOOP-14902
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.4.0
>Reporter: Jason Lowe
>Assignee: Hanisha Koneru
> Fix For: 2.9.0, 2.8.3, 2.7.5, 3.0.0
>
> Attachments: HADOOP-14902.001.patch, HADOOP-14902.002.patch,
> HADOOP-14902.003.patch
>
>
> LoadGenerator#genFile's write close timing code looks like the following:
> {code}
> startTime = Time.now();
> executionTime[WRITE_CLOSE] += (Time.now() - startTime);
> {code}
> That code will generate a zero (or near zero) write close timing since it
> isn't actually closing the file in-between timestamp lookups.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14902) LoadGenerator#genFile write close timing is incorrectly calculated
[
https://issues.apache.org/jira/browse/HADOOP-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184911#comment-16184911
]
Jason Lowe commented on HADOOP-14902:
-
Thanks for updating the patch!
+1 lgtm. Committing this.
> LoadGenerator#genFile write close timing is incorrectly calculated
> --
>
> Key: HADOOP-14902
> URL: https://issues.apache.org/jira/browse/HADOOP-14902
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.4.0
>Reporter: Jason Lowe
>Assignee: Hanisha Koneru
> Attachments: HADOOP-14902.001.patch, HADOOP-14902.002.patch,
> HADOOP-14902.003.patch
>
>
> LoadGenerator#genFile's write close timing code looks like the following:
> {code}
> startTime = Time.now();
> executionTime[WRITE_CLOSE] += (Time.now() - startTime);
> {code}
> That code will generate a zero (or near zero) write close timing since it
> isn't actually closing the file in-between timestamp lookups.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14902) LoadGenerator#genFile write close timing is incorrectly calculated
[
https://issues.apache.org/jira/browse/HADOOP-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183447#comment-16183447
]
Hadoop QA commented on HADOOP-14902:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 4s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 23s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestKDiag |
| | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14902 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12889380/HADOOP-14902.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux cb5f2495e04e 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c87db8d |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13392/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13392/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13392/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> LoadGenerator#genFile write close
[jira] [Commented] (HADOOP-14902) LoadGenerator#genFile write close timing is incorrectly calculated
[
https://issues.apache.org/jira/browse/HADOOP-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182607#comment-16182607
]
Jason Lowe commented on HADOOP-14902:
-
Thanks for updating the patch! There's a potential NPE if the fc.create throws
and leaves {{out}} null where the finally block will try to close it. The
fc.create call should be moved outside of the try block.
> LoadGenerator#genFile write close timing is incorrectly calculated
> --
>
> Key: HADOOP-14902
> URL: https://issues.apache.org/jira/browse/HADOOP-14902
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.4.0
>Reporter: Jason Lowe
>Assignee: Hanisha Koneru
> Attachments: HADOOP-14902.001.patch, HADOOP-14902.002.patch
>
>
> LoadGenerator#genFile's write close timing code looks like the following:
> {code}
> startTime = Time.now();
> executionTime[WRITE_CLOSE] += (Time.now() - startTime);
> {code}
> That code will generate a zero (or near zero) write close timing since it
> isn't actually closing the file in-between timestamp lookups.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14902) LoadGenerator#genFile write close timing is incorrectly calculated
[
https://issues.apache.org/jira/browse/HADOOP-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181663#comment-16181663
]
Jason Lowe commented on HADOOP-14902:
-
Thanks for the patch!
Since genFile already throws IOExceptions for write errors, it seems incorrect
to suppress errors encountered during close. IMHO they should be treated the
same, otherwise callers of genFIle may believe the file was written properly
when it wasn't. Therefore I think we can simplify it a bit where we don't need
a nested try block. All we need to do is track whether the file was closed
within the try block and have the finally block close it if necessary with a
straight out.close(). The exception can propagate out just as it would for a
write error.
> LoadGenerator#genFile write close timing is incorrectly calculated
> --
>
> Key: HADOOP-14902
> URL: https://issues.apache.org/jira/browse/HADOOP-14902
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.4.0
>Reporter: Jason Lowe
>Assignee: Hanisha Koneru
> Attachments: HADOOP-14902.001.patch
>
>
> LoadGenerator#genFile's write close timing code looks like the following:
> {code}
> startTime = Time.now();
> executionTime[WRITE_CLOSE] += (Time.now() - startTime);
> {code}
> That code will generate a zero (or near zero) write close timing since it
> isn't actually closing the file in-between timestamp lookups.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14902) LoadGenerator#genFile write close timing is incorrectly calculated
[
https://issues.apache.org/jira/browse/HADOOP-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181635#comment-16181635
]
Hadoop QA commented on HADOOP-14902:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 50s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
24s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
46s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 87m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HADOOP-14902 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12889141/HADOOP-14902.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux ea4fc528bfce 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9df0500 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13383/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13383/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> LoadGenerator#genFile write close timing is incorrectly calculated
> --
>
> Key: HADOOP-14902
> URL: https://issues.apache.org/jira/browse/HADOOP-14902
> Project: Hadoop
[jira] [Commented] (HADOOP-14902) LoadGenerator#genFile write close timing is incorrectly calculated
[
https://issues.apache.org/jira/browse/HADOOP-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177115#comment-16177115
]
Hanisha Koneru commented on HADOOP-14902:
-
Thanks for reporting this bug, [~jlowe].
As per your
[comment|https://issues.apache.org/jira/browse/HADOOP-14881?focusedCommentId=16177074=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16177074]
in HADOOP-14881, I agree that we should not do double close as a norm.
And I think we should not update the metric if close fails. The metric tracks
the execution time for close operation. If we add the time when exception is
thrown, it would pollute the metric.
What do you think?
> LoadGenerator#genFile write close timing is incorrectly calculated
> --
>
> Key: HADOOP-14902
> URL: https://issues.apache.org/jira/browse/HADOOP-14902
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.4.0
>Reporter: Jason Lowe
>Assignee: Hanisha Koneru
>
> LoadGenerator#genFile's write close timing code looks like the following:
> {code}
> startTime = Time.now();
> executionTime[WRITE_CLOSE] += (Time.now() - startTime);
> {code}
> That code will generate a zero (or near zero) write close timing since it
> isn't actually closing the file in-between timestamp lookups.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
