[
https://issues.apache.org/jira/browse/HADOOP-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991891#comment-15991891
]
Junping Du commented on HADOOP-14284:
-------------------------------------
bq. I'm pretty sure unit tests still dominate the build time.
I mean skip the unit test.
bq. After that, I wouldn't be surprised if it's YARN.
>From my local report for mvn install, no evidence to show YARN have any
>relationship with build slowness. All shading related components are
>dominating the build speed:
{noformat}
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.324 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 2.295 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 0.542 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.355 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 0.111 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.150 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 2.337 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 0.662 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.585 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 2.044 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 27.720 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 2.067 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 3.101 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.075 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 21.961 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 27.508 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 0.475 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 16.516 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 1.081 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.049 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.052 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 8.417 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 11.146 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.064 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 3.441 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 5.758 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 1.541 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 2.741 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 2.264 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 14.351 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 2.159 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 3.087 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 1.686 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 2.140 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 3.577 s]
[INFO] Apache Hadoop YARN Timeline Service HBase tests .... SUCCESS [ 4.854 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.083 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 1.650 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.062 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.085 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 2.026 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.103 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 2.950 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.269 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 7.138 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 4.366 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 1.704 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 4.262 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 2.684 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 6.478 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.172 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [ 2.150 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 2.053 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 0.290 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 1.686 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 2.631 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 0.953 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.250 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 1.696 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 1.670 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 0.813 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 0.935 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 0.071 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 1.317 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 2.900 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [ 1.079 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 2.465 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 1.354 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [ 3.009 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 2.571 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 4.093 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 1.597 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 1.670 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.060 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [02:29 min]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [01:11 min]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [ 0.275 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [01:59 min]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [ 0.190 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [ 0.224 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 0.253 s]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [ 0.053 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.970 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.039 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:02 min
[INFO] Finished at: 2017-05-01T17:09:55-07:00
[INFO] Final Memory: 143M/500M
[INFO] ------------------------------------------------------------------------
{noformat}
Comparing with branch-2, build time for "Apache Hadoop Client" are increasing
from 2+ s to several minutes (for API, Runtime, etc.). Also, build of
Minicluster are changing from 1.3 s to 2 minutes, etc. Still negligible change
for build process?
> Shade Guava everywhere
> ----------------------
>
> Key: HADOOP-14284
> URL: https://issues.apache.org/jira/browse/HADOOP-14284
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Affects Versions: 3.0.0-alpha3
> Reporter: Andrew Wang
> Assignee: Tsuyoshi Ozawa
> Priority: Blocker
> Attachments: HADOOP-14238.pre001.patch, HADOOP-14284.002.patch,
> HADOOP-14284.004.patch, HADOOP-14284.007.patch, HADOOP-14284.010.patch
>
>
> HADOOP-10101 upgraded the guava version for 3.x to 21.
> Guava is broadly used by Java projects that consume our artifacts.
> Unfortunately, these projects also consume our private artifacts like
> {{hadoop-hdfs}}. They also are unlikely on the new shaded client introduced
> by HADOOP-11804, currently only available in 3.0.0-alpha2.
> We should shade Guava everywhere to proactively avoid breaking downstreams.
> This isn't a requirement for all dependency upgrades, but it's necessary for
> known-bad dependencies like Guava.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
