[jira] [Updated] (HADOOP-15124) Slow FileSystem.Statistics counters implementation
[ https://issues.apache.org/jira/browse/HADOOP-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated HADOOP-15124: --- Status: Open (was: Patch Available) > Slow FileSystem.Statistics counters implementation > -- > > Key: HADOOP-15124 > URL: https://issues.apache.org/jira/browse/HADOOP-15124 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.0.0, 2.7.5, 2.8.3, 2.9.0, 3.1.0 >Reporter: Igor Dvorzhak >Assignee: Igor Dvorzhak >Priority: Major > Labels: common, filesystem, fs, statistics > Attachments: HADOOP-15124.001.patch > > > While profiling 1TB TeraGen job on Hadoop 2.8.2 cluster (Google Dataproc, 2 > workers, GCS connector) I saw that FileSystem.Statistics code paths Wall time > is 5.58% and CPU time is 26.5% of total execution time. > After switching FileSystem.Statistics implementation to LongAdder, consumed > Wall time decreased to 0.006% and CPU time to 0.104% of total execution time. > Total job runtime decreased from 66 mins to 61 mins. > These results are not conclusive, because I didn't benchmark multiple times > to average results, but regardless of performance gains switching to > LongAdder simplifies code and reduces its complexity. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15124) Slow FileSystem.Statistics counters implementation
[ https://issues.apache.org/jira/browse/HADOOP-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated HADOOP-15124: --- Status: Patch Available (was: Open) > Slow FileSystem.Statistics counters implementation > -- > > Key: HADOOP-15124 > URL: https://issues.apache.org/jira/browse/HADOOP-15124 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 3.0.0, 2.7.5, 2.8.3, 2.9.0, 3.1.0 >Reporter: Igor Dvorzhak >Assignee: Igor Dvorzhak >Priority: Major > Labels: common, filesystem, fs, statistics > Attachments: HADOOP-15124.001.patch > > > While profiling 1TB TeraGen job on Hadoop 2.8.2 cluster (Google Dataproc, 2 > workers, GCS connector) I saw that FileSystem.Statistics code paths Wall time > is 5.58% and CPU time is 26.5% of total execution time. > After switching FileSystem.Statistics implementation to LongAdder, consumed > Wall time decreased to 0.006% and CPU time to 0.104% of total execution time. > Total job runtime decreased from 66 mins to 61 mins. > These results are not conclusive, because I didn't benchmark multiple times > to average results, but regardless of performance gains switching to > LongAdder simplifies code and reduces its complexity. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15124) Slow FileSystem.Statistics counters implementation
[ https://issues.apache.org/jira/browse/HADOOP-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated HADOOP-15124: --- Affects Version/s: 3.1.0 > Slow FileSystem.Statistics counters implementation > -- > > Key: HADOOP-15124 > URL: https://issues.apache.org/jira/browse/HADOOP-15124 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0, 3.1.0 >Reporter: Igor Dvorzhak >Assignee: Igor Dvorzhak >Priority: Major > Labels: common, filesystem, fs, statistics > Attachments: HADOOP-15124.001.patch > > > While profiling 1TB TeraGen job on Hadoop 2.8.2 cluster (Google Dataproc, 2 > workers, GCS connector) I saw that FileSystem.Statistics code paths Wall time > is 5.58% and CPU time is 26.5% of total execution time. > After switching FileSystem.Statistics implementation to LongAdder, consumed > Wall time decreased to 0.006% and CPU time to 0.104% of total execution time. > Total job runtime decreased from 66 mins to 61 mins. > These results are not conclusive, because I didn't benchmark multiple times > to average results, but regardless of performance gains switching to > LongAdder simplifies code and reduces its complexity. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15124) Slow FileSystem.Statistics counters implementation
[ https://issues.apache.org/jira/browse/HADOOP-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated HADOOP-15124: --- Attachment: HADOOP-15124.001.patch > Slow FileSystem.Statistics counters implementation > -- > > Key: HADOOP-15124 > URL: https://issues.apache.org/jira/browse/HADOOP-15124 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0 >Reporter: Igor Dvorzhak >Assignee: Igor Dvorzhak >Priority: Major > Labels: common, filesystem, fs, statistics > Attachments: HADOOP-15124.001.patch > > > While profiling 1TB TeraGen job on Hadoop 2.8.2 cluster (Google Dataproc, 2 > workers, GCS connector) I saw that FileSystem.Statistics code paths Wall time > is 5.58% and CPU time is 26.5% of total execution time. > After switching FileSystem.Statistics implementation to LongAdder, consumed > Wall time decreased to 0.006% and CPU time to 0.104% of total execution time. > Total job runtime decreased from 66 mins to 61 mins. > These results are not conclusive, because I didn't benchmark multiple times > to average results, but regardless of performance gains switching to > LongAdder simplifies code and reduces its complexity. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15124) Slow FileSystem.Statistics counters implementation
[ https://issues.apache.org/jira/browse/HADOOP-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated HADOOP-15124: --- Attachment: (was: HADOOP-15124.001.patch) > Slow FileSystem.Statistics counters implementation > -- > > Key: HADOOP-15124 > URL: https://issues.apache.org/jira/browse/HADOOP-15124 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0 >Reporter: Igor Dvorzhak >Assignee: Igor Dvorzhak >Priority: Major > Labels: common, filesystem, fs, statistics > Attachments: HADOOP-15124.001.patch > > > While profiling 1TB TeraGen job on Hadoop 2.8.2 cluster (Google Dataproc, 2 > workers, GCS connector) I saw that FileSystem.Statistics code paths Wall time > is 5.58% and CPU time is 26.5% of total execution time. > After switching FileSystem.Statistics implementation to LongAdder, consumed > Wall time decreased to 0.006% and CPU time to 0.104% of total execution time. > Total job runtime decreased from 66 mins to 61 mins. > These results are not conclusive, because I didn't benchmark multiple times > to average results, but regardless of performance gains switching to > LongAdder simplifies code and reduces its complexity. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15124) Slow FileSystem.Statistics counters implementation
[ https://issues.apache.org/jira/browse/HADOOP-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated HADOOP-15124: --- Target Version/s: 3.2.0 (was: 3.1.0) > Slow FileSystem.Statistics counters implementation > -- > > Key: HADOOP-15124 > URL: https://issues.apache.org/jira/browse/HADOOP-15124 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0 >Reporter: Igor Dvorzhak >Assignee: Igor Dvorzhak >Priority: Major > Labels: common, filesystem, fs, statistics > Attachments: HADOOP-15124.001.patch > > > While profiling 1TB TeraGen job on Hadoop 2.8.2 cluster (Google Dataproc, 2 > workers, GCS connector) I saw that FileSystem.Statistics code paths Wall time > is 5.58% and CPU time is 26.5% of total execution time. > After switching FileSystem.Statistics implementation to LongAdder, consumed > Wall time decreased to 0.006% and CPU time to 0.104% of total execution time. > Total job runtime decreased from 66 mins to 61 mins. > These results are not conclusive, because I didn't benchmark multiple times > to average results, but regardless of performance gains switching to > LongAdder simplifies code and reduces its complexity. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15124) Slow FileSystem.Statistics counters implementation
[
https://issues.apache.org/jira/browse/HADOOP-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520952#comment-16520952
]
genericqa commented on HADOOP-15124:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HADOOP-15124 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15124 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909637/HADOOP-15124.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14810/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Slow FileSystem.Statistics counters implementation
> --
>
> Key: HADOOP-15124
> URL: https://issues.apache.org/jira/browse/HADOOP-15124
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: common
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0
>Reporter: Igor Dvorzhak
>Assignee: Igor Dvorzhak
>Priority: Major
> Labels: common, filesystem, fs, statistics
> Attachments: HADOOP-15124.001.patch
>
>
> While profiling 1TB TeraGen job on Hadoop 2.8.2 cluster (Google Dataproc, 2
> workers, GCS connector) I saw that FileSystem.Statistics code paths Wall time
> is 5.58% and CPU time is 26.5% of total execution time.
> After switching FileSystem.Statistics implementation to LongAdder, consumed
> Wall time decreased to 0.006% and CPU time to 0.104% of total execution time.
> Total job runtime decreased from 66 mins to 61 mins.
> These results are not conclusive, because I didn't benchmark multiple times
> to average results, but regardless of performance gains switching to
> LongAdder simplifies code and reduces its complexity.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15124) Slow FileSystem.Statistics counters implementation
[ https://issues.apache.org/jira/browse/HADOOP-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Dvorzhak updated HADOOP-15124: --- Target Version/s: 3.1.0 (was: 3.0.2) > Slow FileSystem.Statistics counters implementation > -- > > Key: HADOOP-15124 > URL: https://issues.apache.org/jira/browse/HADOOP-15124 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0 >Reporter: Igor Dvorzhak >Assignee: Igor Dvorzhak >Priority: Major > Labels: common, filesystem, fs, statistics > Attachments: HADOOP-15124.001.patch > > > While profiling 1TB TeraGen job on Hadoop 2.8.2 cluster (Google Dataproc, 2 > workers, GCS connector) I saw that FileSystem.Statistics code paths Wall time > is 5.58% and CPU time is 26.5% of total execution time. > After switching FileSystem.Statistics implementation to LongAdder, consumed > Wall time decreased to 0.006% and CPU time to 0.104% of total execution time. > Total job runtime decreased from 66 mins to 61 mins. > These results are not conclusive, because I didn't benchmark multiple times > to average results, but regardless of performance gains switching to > LongAdder simplifies code and reduces its complexity. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15552) Continue HADOOP-14296 - Move logging APIs over to slf4j in hadoop-tools
[
https://issues.apache.org/jira/browse/HADOOP-15552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520896#comment-16520896
]
genericqa commented on HADOOP-15552:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 33 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
25s{color} | {color:green} root generated 0 new + 1557 unchanged - 6 fixed =
1557 total (was 1563) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 11m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
49s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
48s{color} | {color:green} hadoop-streaming in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
18s{color} | {color:green} hadoop-distcp in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
8s{color} | {color:green} hadoop-archives in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
50s{color} | {color:green} hadoop-archive-logs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} hadoop-rumen in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
51s{color} | {color:green} hadoop-gridmix in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-datajoin in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
56s{color} | {color:green} hadoop-extras in the patch
[jira] [Commented] (HADOOP-15423) Merge fileCache and dirCache into one single cache in LocalMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-15423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520841#comment-16520841
]
Sean Mackrory commented on HADOOP-15423:
+1, but as long as we're rewriting that metadata structure, can we add an
assertion that the same instance is never used to store both directory metadata
and file metadata? I'd always felt a bit uncomfortable about that not being an
explicit impossibility.
> Merge fileCache and dirCache into one single cache in LocalMetadataStore
>
>
> Key: HADOOP-15423
> URL: https://issues.apache.org/jira/browse/HADOOP-15423
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Minor
> Attachments: HADOOP-15423.001.patch
>
>
> Right now the s3guard.LocalMetadataStore uses two HashMap in the
> implementation - one for the file and one for the dir hash.
> {code:java}
> /** Contains directories and files. */
> private Cache fileCache;
> /** Contains directory listings. */
> private Cache dirCache;
> {code}
> It would be nice to have only one hash instead of these two for storing the
> values. An idea for the implementation would be to have a class with nullable
> fields:
> {code:java}
> static class LocalMetaEntry {
> @Nullable
> public PathMetadata pathMetadata;
> @Nullable
> public DirListingMetadata dirListingMetadata;
> }
> {code}
> or a Pair (tuple):
> {code:java}
> Pair metaEntry;
> {code}
> And only one hash/cache for these elements.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14891) Remove references to Guava Objects.toStringHelper
[
https://issues.apache.org/jira/browse/HADOOP-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520830#comment-16520830
]
Jonathan Hung commented on HADOOP-14891:
Thanks Konstantin. I have committed this to branch-2.7.
> Remove references to Guava Objects.toStringHelper
> -
>
> Key: HADOOP-14891
> URL: https://issues.apache.org/jira/browse/HADOOP-14891
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.0, 2.8.1, 2.7.8
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Fix For: 2.9.0, 2.8.3, 2.7.8
>
> Attachments: HADOOP-14891.001-branch-2.patch
>
>
> Use provided a guava 23.0 jar as part of the job submission.
> {code}
> 2017-09-20 16:10:42,897 [INFO] [main] |service.AbstractService|: Service
> org.apache.tez.dag.app.DAGAppMaster failed in state STARTED; cause:
> org.apache.hadoop.service.ServiceStateException: java.lang.NoSuchMethodError:
> com.google.common.base.Objects.toStringHelper(Ljava/lang/Object;)Lcom/google/common/base/Objects$ToStringHelper;
> org.apache.hadoop.service.ServiceStateException: java.lang.NoSuchMethodError:
> com.google.common.base.Objects.toStringHelper(Ljava/lang/Object;)Lcom/google/common/base/Objects$ToStringHelper;
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
> at
> org.apache.tez.dag.app.DAGAppMaster.startServices(DAGAppMaster.java:1989)
> at
> org.apache.tez.dag.app.DAGAppMaster.serviceStart(DAGAppMaster.java:2056)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2707)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1936)
> at
> org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2703)
> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2508)
> Caused by: java.lang.NoSuchMethodError:
> com.google.common.base.Objects.toStringHelper(Ljava/lang/Object;)Lcom/google/common/base/Objects$ToStringHelper;
> at
> org.apache.hadoop.metrics2.lib.MetricsRegistry.toString(MetricsRegistry.java:419)
> at java.lang.String.valueOf(String.java:2994)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at org.apache.hadoop.ipc.metrics.RpcMetrics.(RpcMetrics.java:74)
> at org.apache.hadoop.ipc.metrics.RpcMetrics.create(RpcMetrics.java:80)
> at org.apache.hadoop.ipc.Server.(Server.java:2658)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:968)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:367)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:342)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:810)
> at
> org.apache.tez.dag.api.client.DAGClientServer.createServer(DAGClientServer.java:134)
> at
> org.apache.tez.dag.api.client.DAGClientServer.serviceStart(DAGClientServer.java:82)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.tez.dag.app.DAGAppMaster$ServiceWithDependency.start(DAGAppMaster.java:1909)
> at
> org.apache.tez.dag.app.DAGAppMaster$ServiceThread.run(DAGAppMaster.java:1930)
> 2017-09-20 16:10:42,898 [ERROR] [main] |rm.TaskSchedulerManager|: Failed to
> do a clean initiateStop for Scheduler: [0:TezYarn]
> {code}
> Metrics2 has been relying on deprecated toStringHelper for some time now
> which was finally removed in guava 21.0. Removing the dependency on this
> method will free up the user to supplying their own guava jar again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14891) Remove references to Guava Objects.toStringHelper
[
https://issues.apache.org/jira/browse/HADOOP-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated HADOOP-14891:
---
Fix Version/s: 2.7.8
> Remove references to Guava Objects.toStringHelper
> -
>
> Key: HADOOP-14891
> URL: https://issues.apache.org/jira/browse/HADOOP-14891
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.0, 2.8.1, 2.7.8
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Fix For: 2.9.0, 2.8.3, 2.7.8
>
> Attachments: HADOOP-14891.001-branch-2.patch
>
>
> Use provided a guava 23.0 jar as part of the job submission.
> {code}
> 2017-09-20 16:10:42,897 [INFO] [main] |service.AbstractService|: Service
> org.apache.tez.dag.app.DAGAppMaster failed in state STARTED; cause:
> org.apache.hadoop.service.ServiceStateException: java.lang.NoSuchMethodError:
> com.google.common.base.Objects.toStringHelper(Ljava/lang/Object;)Lcom/google/common/base/Objects$ToStringHelper;
> org.apache.hadoop.service.ServiceStateException: java.lang.NoSuchMethodError:
> com.google.common.base.Objects.toStringHelper(Ljava/lang/Object;)Lcom/google/common/base/Objects$ToStringHelper;
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
> at
> org.apache.tez.dag.app.DAGAppMaster.startServices(DAGAppMaster.java:1989)
> at
> org.apache.tez.dag.app.DAGAppMaster.serviceStart(DAGAppMaster.java:2056)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2707)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1936)
> at
> org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2703)
> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2508)
> Caused by: java.lang.NoSuchMethodError:
> com.google.common.base.Objects.toStringHelper(Ljava/lang/Object;)Lcom/google/common/base/Objects$ToStringHelper;
> at
> org.apache.hadoop.metrics2.lib.MetricsRegistry.toString(MetricsRegistry.java:419)
> at java.lang.String.valueOf(String.java:2994)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at org.apache.hadoop.ipc.metrics.RpcMetrics.(RpcMetrics.java:74)
> at org.apache.hadoop.ipc.metrics.RpcMetrics.create(RpcMetrics.java:80)
> at org.apache.hadoop.ipc.Server.(Server.java:2658)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:968)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:367)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:342)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:810)
> at
> org.apache.tez.dag.api.client.DAGClientServer.createServer(DAGClientServer.java:134)
> at
> org.apache.tez.dag.api.client.DAGClientServer.serviceStart(DAGClientServer.java:82)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.tez.dag.app.DAGAppMaster$ServiceWithDependency.start(DAGAppMaster.java:1909)
> at
> org.apache.tez.dag.app.DAGAppMaster$ServiceThread.run(DAGAppMaster.java:1930)
> 2017-09-20 16:10:42,898 [ERROR] [main] |rm.TaskSchedulerManager|: Failed to
> do a clean initiateStop for Scheduler: [0:TezYarn]
> {code}
> Metrics2 has been relying on deprecated toStringHelper for some time now
> which was finally removed in guava 21.0. Removing the dependency on this
> method will free up the user to supplying their own guava jar again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520816#comment-16520816
]
genericqa commented on HADOOP-14396:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
18s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}123m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-14396 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928806/HADOOP-14396.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 1407113c992e 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 55fad6a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14809/testReport/ |
| Max. process+thread count | 1520 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14809/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add builder interface to FileContext
>
>
> Key: HADOOP-14396
> URL:
[jira] [Commented] (HADOOP-14891) Remove references to Guava Objects.toStringHelper
[
https://issues.apache.org/jira/browse/HADOOP-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520797#comment-16520797
]
Konstantin Shvachko commented on HADOOP-14891:
--
+1 Makes sense to backport to branch-2.7.
Seems to be applying cleanly.
If I were looking at it from the start I would've suggested to use the original
chain call pattern, like {{sb.append(..).append(..)}}.
May be something for the future.
> Remove references to Guava Objects.toStringHelper
> -
>
> Key: HADOOP-14891
> URL: https://issues.apache.org/jira/browse/HADOOP-14891
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.9.0, 2.8.1, 2.7.8
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Fix For: 2.9.0, 2.8.3
>
> Attachments: HADOOP-14891.001-branch-2.patch
>
>
> Use provided a guava 23.0 jar as part of the job submission.
> {code}
> 2017-09-20 16:10:42,897 [INFO] [main] |service.AbstractService|: Service
> org.apache.tez.dag.app.DAGAppMaster failed in state STARTED; cause:
> org.apache.hadoop.service.ServiceStateException: java.lang.NoSuchMethodError:
> com.google.common.base.Objects.toStringHelper(Ljava/lang/Object;)Lcom/google/common/base/Objects$ToStringHelper;
> org.apache.hadoop.service.ServiceStateException: java.lang.NoSuchMethodError:
> com.google.common.base.Objects.toStringHelper(Ljava/lang/Object;)Lcom/google/common/base/Objects$ToStringHelper;
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
> at
> org.apache.tez.dag.app.DAGAppMaster.startServices(DAGAppMaster.java:1989)
> at
> org.apache.tez.dag.app.DAGAppMaster.serviceStart(DAGAppMaster.java:2056)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2707)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1936)
> at
> org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2703)
> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2508)
> Caused by: java.lang.NoSuchMethodError:
> com.google.common.base.Objects.toStringHelper(Ljava/lang/Object;)Lcom/google/common/base/Objects$ToStringHelper;
> at
> org.apache.hadoop.metrics2.lib.MetricsRegistry.toString(MetricsRegistry.java:419)
> at java.lang.String.valueOf(String.java:2994)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at org.apache.hadoop.ipc.metrics.RpcMetrics.(RpcMetrics.java:74)
> at org.apache.hadoop.ipc.metrics.RpcMetrics.create(RpcMetrics.java:80)
> at org.apache.hadoop.ipc.Server.(Server.java:2658)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:968)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:367)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:342)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:810)
> at
> org.apache.tez.dag.api.client.DAGClientServer.createServer(DAGClientServer.java:134)
> at
> org.apache.tez.dag.api.client.DAGClientServer.serviceStart(DAGClientServer.java:82)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.tez.dag.app.DAGAppMaster$ServiceWithDependency.start(DAGAppMaster.java:1909)
> at
> org.apache.tez.dag.app.DAGAppMaster$ServiceThread.run(DAGAppMaster.java:1930)
> 2017-09-20 16:10:42,898 [ERROR] [main] |rm.TaskSchedulerManager|: Failed to
> do a clean initiateStop for Scheduler: [0:TezYarn]
> {code}
> Metrics2 has been relying on deprecated toStringHelper for some time now
> which was finally removed in guava 21.0. Removing the dependency on this
> method will free up the user to supplying their own guava jar again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15483) Upgrade jquery to version 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520755#comment-16520755 ] Jitendra Nath Pandey commented on HADOOP-15483: --- I am +1 to backport this to 3.1. > Upgrade jquery to version 3.3.1 > --- > > Key: HADOOP-15483 > URL: https://issues.apache.org/jira/browse/HADOOP-15483 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15483.001.patch, HADOOP-15483.002.patch, > HADOOP-15483.003.patch, HADOOP-15483.004.patch, HADOOP-15483.005.patch, > HADOOP-15483.006.patch, HADOOP-15483.007.patch, HADOOP-15483.008.patch > > > This Jira aims to upgrade jquery to version 3.3.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15550) Avoid static initialization of ObjectMappers
[
https://issues.apache.org/jira/browse/HADOOP-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520740#comment-16520740
]
Xiao Chen commented on HADOOP-15550:
Thanks Todd and Steve. Latest patch looks great!
Sorry for the nit, {{WRITER}} and {{MAP_READER}} could be final (seem to recall
checkstyle should complain that, not sure why it doesn't this time). +1 from me
pending that.
> Avoid static initialization of ObjectMappers
>
>
> Key: HADOOP-15550
> URL: https://issues.apache.org/jira/browse/HADOOP-15550
> Project: Hadoop Common
> Issue Type: Bug
> Components: performance
>Affects Versions: 3.2.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Minor
> Attachments: hadoop-15550.txt, hadoop-15550.txt, hadoop-15550.txt,
> hadoop-15550.txt
>
>
> Various classes statically initialize an ObjectMapper READER instance. This
> ends up doing a bunch of class-loading of Jackson libraries that can add up
> to a fair amount of CPU, even if the reader ends up not being used. This is
> particularly the case with WebHdfsFileSystem, which is class-loaded by a
> serviceloader even when unused in a particular job. We should lazy-init these
> members instead of doing so as a static class member.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14396:
Attachment: HADOOP-14396.02.patch
> Add builder interface to FileContext
>
>
> Key: HADOOP-14396
> URL: https://issues.apache.org/jira/browse/HADOOP-14396
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Major
> Attachments: HADOOP-14396.00.patch, HADOOP-14396.01.patch,
> HADOOP-14396.02.patch, HADOOP-14396.02.patch
>
>
> Add builder interface for {{FileContext#create}} and {{FileContext#append}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14396) Add builder interface to FileContext
[
https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520728#comment-16520728
]
Steve Loughran commented on HADOOP-14396:
-
sorry, missed that this was needing its final review
+1 pending yetus being able to merge the latest patch
> Add builder interface to FileContext
>
>
> Key: HADOOP-14396
> URL: https://issues.apache.org/jira/browse/HADOOP-14396
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 2.9.0, 3.0.0-alpha3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Major
> Attachments: HADOOP-14396.00.patch, HADOOP-14396.01.patch,
> HADOOP-14396.02.patch, HADOOP-14396.02.patch
>
>
> Add builder interface for {{FileContext#create}} and {{FileContext#append}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15483) Upgrade jquery to version 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520709#comment-16520709 ] Sunil Govindan commented on HADOOP-15483: - This is very useful change and I think we can backport this to 3.1 as well. [~jnp] [~vinodkv] [~wangda] Thoughts? > Upgrade jquery to version 3.3.1 > --- > > Key: HADOOP-15483 > URL: https://issues.apache.org/jira/browse/HADOOP-15483 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15483.001.patch, HADOOP-15483.002.patch, > HADOOP-15483.003.patch, HADOOP-15483.004.patch, HADOOP-15483.005.patch, > HADOOP-15483.006.patch, HADOOP-15483.007.patch, HADOOP-15483.008.patch > > > This Jira aims to upgrade jquery to version 3.3.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15550) Avoid static initialization of ObjectMappers
[
https://issues.apache.org/jira/browse/HADOOP-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520702#comment-16520702
]
genericqa commented on HADOOP-15550:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
3s{color} | {color:blue} The patch file was not named according to hadoop's
naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute
for instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 33m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 34m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 51s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
49s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
1s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
38s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
45s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
12s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |
[jira] [Created] (HADOOP-15557) Crypto streams should not crash when mis-used
Todd Lipcon created HADOOP-15557: Summary: Crypto streams should not crash when mis-used Key: HADOOP-15557 URL: https://issues.apache.org/jira/browse/HADOOP-15557 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 3.2.0 Reporter: Todd Lipcon In general, the non-positional read APIs for streams in Hadoop Common are meant to be used by only a single thread at a time. It would not make much sense to have concurrent multi-threaded access to seek+read because they modify the stream's file position. Multi-threaded access on input streams can be done using positional read APIs. Multi-threaded access on output streams probably never makes sense. In the case of DFSInputStream, the positional read APIs are marked synchronized, so that even when misused, no strange exceptions are thrown. The results are just somewhat undefined in that it's hard for a thread to know which position was read from. However, when running on an encrypted file system, the results are much worse: since CryptoInputStream's read methods are not marked synchronized, the caller can get strange ByteBuffer exceptions or even a JVM crash due to concurrent use and free of underlying OpenSSL Cipher buffers. The crypto stream wrappers should be made more resilient to such misuse, for example by: (a) making the read methods safer by making them synchronized (so they have the same behavior as DFSInputStream) or (b) trying to detect concurrent access to these methods and throwing ConcurrentModificationException so that the user is alerted to their probable misuse. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15552) Continue HADOOP-14296 - Move logging APIs over to slf4j in hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-15552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520686#comment-16520686 ] Ian Pickering commented on HADOOP-15552: Thanks [~ste...@apache.org] for the comment. I've attached V1 of the patch. > Continue HADOOP-14296 - Move logging APIs over to slf4j in hadoop-tools > --- > > Key: HADOOP-15552 > URL: https://issues.apache.org/jira/browse/HADOOP-15552 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Ian Pickering >Priority: Major > Attachments: HADOOP-15552.v1.patch > > > Some classes in Hadoop-tools were not moved to slf4j > e.g. AliyunOSSInputStream.java, HadoopArchiveLogs.java, > HadoopArchiveLogsRunner.java -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15552) Continue HADOOP-14296 - Move logging APIs over to slf4j in hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-15552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated HADOOP-15552: -- Assignee: Ian Pickering Status: Patch Available (was: Open) > Continue HADOOP-14296 - Move logging APIs over to slf4j in hadoop-tools > --- > > Key: HADOOP-15552 > URL: https://issues.apache.org/jira/browse/HADOOP-15552 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Ian Pickering >Priority: Major > Attachments: HADOOP-15552.v1.patch > > > Some classes in Hadoop-tools were not moved to slf4j > e.g. AliyunOSSInputStream.java, HadoopArchiveLogs.java, > HadoopArchiveLogsRunner.java -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15552) Continue HADOOP-14296 - Move logging APIs over to slf4j in hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-15552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ian Pickering updated HADOOP-15552: --- Attachment: HADOOP-15552.v1.patch > Continue HADOOP-14296 - Move logging APIs over to slf4j in hadoop-tools > --- > > Key: HADOOP-15552 > URL: https://issues.apache.org/jira/browse/HADOOP-15552 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Priority: Major > Attachments: HADOOP-15552.v1.patch > > > Some classes in Hadoop-tools were not moved to slf4j > e.g. AliyunOSSInputStream.java, HadoopArchiveLogs.java, > HadoopArchiveLogsRunner.java -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15416) s3guard diff assert failure if source path not found
[ https://issues.apache.org/jira/browse/HADOOP-15416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520644#comment-16520644 ] Hudson commented on HADOOP-15416: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14464 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14464/]) HADOOP-15416. Clear error message in S3Guard diff if source not found. (mackrorysd: rev 55fad6a3de3125d9e7e2e9a5f8fa5b1b22a1de60) * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java > s3guard diff assert failure if source path not found > > > Key: HADOOP-15416 > URL: https://issues.apache.org/jira/browse/HADOOP-15416 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 > Environment: s3a with fault injection turned on >Reporter: Steve Loughran >Assignee: Gabor Bota >Priority: Minor > Attachments: HADOOP-15416.001.patch > > > Got an illegal argument exception trying to do a s3guard diff in a test run. > Underlying cause: directory in supplied s3a path didn't exist -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15536) Adding support in FileUtil for the creation of directories
[ https://issues.apache.org/jira/browse/HADOOP-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520618#comment-16520618 ] Giovanni Matteo Fumarola commented on HADOOP-15536: --- [~ste...@apache.org] can you take a look? > Adding support in FileUtil for the creation of directories > -- > > Key: HADOOP-15536 > URL: https://issues.apache.org/jira/browse/HADOOP-15536 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: HADOOP-15536-HADOOP-15461.v1.patch, > HADOOP-15536-HADOOP-15461.v2.patch, HADOOP-15536-HADOOP-15461.v3.patch, > HADOOP-15536.v1.patch > > > Adding support in FileUtil for the creation of directories. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15550) Avoid static initialization of ObjectMappers
[ https://issues.apache.org/jira/browse/HADOOP-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HADOOP-15550: - Attachment: hadoop-15550.txt > Avoid static initialization of ObjectMappers > > > Key: HADOOP-15550 > URL: https://issues.apache.org/jira/browse/HADOOP-15550 > Project: Hadoop Common > Issue Type: Bug > Components: performance >Affects Versions: 3.2.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: hadoop-15550.txt, hadoop-15550.txt, hadoop-15550.txt, > hadoop-15550.txt > > > Various classes statically initialize an ObjectMapper READER instance. This > ends up doing a bunch of class-loading of Jackson libraries that can add up > to a fair amount of CPU, even if the reader ends up not being used. This is > particularly the case with WebHdfsFileSystem, which is class-loaded by a > serviceloader even when unused in a particular job. We should lazy-init these > members instead of doing so as a static class member. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15548) Randomize local dirs
[ https://issues.apache.org/jira/browse/HADOOP-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520320#comment-16520320 ] Jim Brennan commented on HADOOP-15548: -- [~eepayne], this one looks like it is ready to review as well. Thanks! > Randomize local dirs > > > Key: HADOOP-15548 > URL: https://issues.apache.org/jira/browse/HADOOP-15548 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Minor > Attachments: HADOOP-15548.001.patch > > > shuffle LOCAL_DIRS, LOG_DIRS and LOCAL_USER_DIRS when launching container. > Some applications will process these in exactly the same way in every > container (e.g. roundrobin) which can cause disks to get unnecessarily > overloaded (e.g. one output file written to first entry specified in the > environment variable). > There are two paths for local dir allocation, depending on whether the size > is unknown or known. The unknown path already uses a random algorithm. The > known path initializes with a random starting point, and then goes > round-robin after that. When selecting a dir, it increments the last used by > one and then checks sequentially until it finds a dir that satisfies the > request. Proposal is to increment by a random value of between 1 and > num_dirs - 1, and then check sequentially from there. This should result in > a more random selection in all cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15554) Improve JIT performance for Configuration parsing
[ https://issues.apache.org/jira/browse/HADOOP-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520299#comment-16520299 ] Jonathan Eagles commented on HADOOP-15554: -- [~tlipcon], thanks for pointing me to this improvement. I'm going to be away from my computer for a few days. If it can wait until Wednesday, I'd be happy to review? > Improve JIT performance for Configuration parsing > - > > Key: HADOOP-15554 > URL: https://issues.apache.org/jira/browse/HADOOP-15554 > Project: Hadoop Common > Issue Type: Improvement > Components: conf, performance >Affects Versions: 3.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: hadoop-15554.patch, hadoop-15554.patch > > > In investigating a performance regression for small tasks between Hadoop 2 > and Hadoop 3, we found that the amount of time spent in JIT was significantly > higher. Using jitwatch we were able to determine that, due to a combination > of switching from DOM to SAX style parsing and just having more configuration > key/value pairs, Configuration.loadResource is now getting compiled with the > C2 compiler and taking quite some time. Breaking that very large function up > into several smaller ones and eliminating some redundant bits of code > improves the JIT performance measurably. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
