Hi Steve, Just tried building with -DskipShade, It worked for me. Executed *mvn clean install -DskipTests -DskipShade -Pdist -Dmaven.javadoc.skip=true*
Sharing the result : [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary for Apache Hadoop Main 3.4.0-SNAPSHOT: [INFO] [INFO] Apache Hadoop Main ................................. SUCCESS [ 1.587 s] [INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 19.332 s] [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 3.538 s] [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 2.756 s] [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.348 s] [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 2.319 s] [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 5.573 s] [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 2.083 s] [INFO] Apache Hadoop Auth ................................. SUCCESS [ 9.620 s] [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 4.134 s] [INFO] Apache Hadoop Common ............................... SUCCESS [ 51.670 s] [INFO] Apache Hadoop NFS .................................. SUCCESS [ 3.019 s] [INFO] Apache Hadoop KMS .................................. SUCCESS [ 3.651 s] [INFO] Apache Hadoop Registry ............................. SUCCESS [ 4.862 s] [INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.109 s] [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 24.093 s] [INFO] Apache Hadoop HDFS ................................. SUCCESS [ 46.701 s] [INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 2.147 s] [INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 4.048 s] [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 3.467 s] [INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 6.077 s] [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.340 s] [INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.349 s] [INFO] Apache Hadoop YARN API ............................. SUCCESS [ 16.439 s] [INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 20.588 s] [INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.082 s] [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 6.682 s] [INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 13.558 s] [INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 1.800 s] [INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 4.442 s] [INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 2.093 s] [INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 19.693 s] [INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 3.153 s] [INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 3.888 s] [INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 2.198 s] [INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 1.360 s] [INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 0.078 s] [INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [ 8.619 s] [INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [ 6.664 s] [INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [ 0.060 s] [INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SUCCESS [ 2.963 s] [INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [ 3.729 s] [INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 2.989 s] [INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [ 2.976 s] [INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.335 s] [INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 2.720 s] [INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.341 s] [INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.223 s] [INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 8.664 s] [INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 5.485 s] [INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 3.390 s] [INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 3.636 s] [INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 3.584 s] [INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 5.087 s] [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 2.747 s] [INFO] Apache Hadoop YARN Services ........................ SUCCESS [ 0.393 s] [INFO] Apache Hadoop YARN Services Core ................... SUCCESS [ 4.652 s] [INFO] Apache Hadoop YARN Services API .................... SUCCESS [ 2.859 s] [INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [ 0.145 s] [INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [ 39.601 s] [INFO] Apache Hadoop YARN Application Catalog Docker Image SUCCESS [ 0.371 s] [INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [ 0.151 s] [INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [ 1.250 s] [INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.131 s] [INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 1.733 s] [INFO] Apache Hadoop YARN UI .............................. SUCCESS [ 0.070 s] [INFO] Apache Hadoop YARN CSI ............................. SUCCESS [ 7.734 s] [INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 7.258 s] [INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 0.684 s] [INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [ 3.412 s] [INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [ 2.193 s] [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 3.257 s] [INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 6.359 s] [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 1.383 s] [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 2.370 s] [INFO] Apache Hadoop Client Aggregator .................... SUCCESS [ 1.699 s] [INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [ 1.390 s] [INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [ 3.769 s] [INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [ 1.027 s] [INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [ 4.113 s] [INFO] Apache Hadoop Dynamometer .......................... SUCCESS [ 0.539 s] [INFO] Apache Hadoop Archives ............................. SUCCESS [ 0.831 s] [INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 0.635 s] [INFO] Apache Hadoop Rumen ................................ SUCCESS [ 3.435 s] [INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 1.090 s] [INFO] Apache Hadoop Data Join ............................ SUCCESS [ 0.530 s] [INFO] Apache Hadoop Extras ............................... SUCCESS [ 0.488 s] [INFO] Apache Hadoop Pipes ................................ SUCCESS [ 0.075 s] [INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 1.449 s] [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 16.265 s] [INFO] Apache Hadoop Kafka Library support ................ SUCCESS [ 1.903 s] [INFO] Apache Hadoop Azure support ........................ SUCCESS [ 4.522 s] [INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 2.036 s] [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 3.199 s] [INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 2.126 s] [INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 0.710 s] [INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [ 0.681 s] [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 4.655 s] [INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.068 s] [INFO] Apache Hadoop Client API ........................... SUCCESS [ 2.329 s] [INFO] Apache Hadoop Client Runtime ....................... SUCCESS [ 0.646 s] [INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [ 1.650 s] [INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [ 2.925 s] [INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [ 0.370 s] [INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [ 0.688 s] [INFO] Apache Hadoop Distribution ......................... SUCCESS [ 41.321 s] [INFO] Apache Hadoop Client Modules ....................... SUCCESS [ 0.663 s] [INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [ 1.865 s] [INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.810 s] [INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.076 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 09:23 min [INFO] Finished at: 2020-05-12T19:46:31+05:30 [INFO] ------------------------------------------------------------------------ ayush@ayushpc:~/trunk$ mvn clean install -DskipTests -DskipShade -Pdist -Dmaven.javadoc.skip=true -Ayush On Tue, 12 May 2020 at 19:21, Steve Loughran <ste...@cloudera.com.invalid> wrote: > every week or so I do a -Pdist build of hadoop, so I can have an up to date > command line. Today I kicked off the build with -Pdist -DskipTests > -Dmaven.javadoc.skip=true > > And it took half an hour to complete -with 1/3 of that time just shading > the minicluster > > [INFO] Apache Hadoop Main ................................. SUCCESS [ > 1.065 s] > [INFO] Apache Hadoop Build Tools .......................... SUCCESS [ > 3.606 s] > [INFO] Apache Hadoop Project POM .......................... SUCCESS [ > 1.374 s] > [INFO] Apache Hadoop Annotations .......................... SUCCESS [ > 1.209 s] > [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ > 0.221 s] > [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ > 1.096 s] > [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ > 4.158 s] > [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ > 0.710 s] > [INFO] Apache Hadoop Auth ................................. SUCCESS [ > 2.969 s] > [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ > 1.591 s] > [INFO] Apache Hadoop Common ............................... SUCCESS [ > 49.545 s] > [INFO] Apache Hadoop NFS .................................. SUCCESS [ > 1.647 s] > [INFO] Apache Hadoop KMS .................................. SUCCESS [ > 3.279 s] > [INFO] Apache Hadoop Registry ............................. SUCCESS [ > 4.420 s] > [INFO] Apache Hadoop Common Project ....................... SUCCESS [ > 0.118 s] > [INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ > 32.316 s] > [INFO] Apache Hadoop HDFS ................................. SUCCESS [01:19 > min] > [INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ > 1.336 s] > [INFO] Apache Hadoop HttpFS ............................... SUCCESS [ > 4.519 s] > [INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ > 2.166 s] > [INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ > 13.665 s] > [INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ > 0.103 s] > [INFO] Apache Hadoop YARN ................................. SUCCESS [ > 0.097 s] > [INFO] Apache Hadoop YARN API ............................. SUCCESS [ > 23.601 s] > [INFO] Apache Hadoop YARN Common .......................... SUCCESS [ > 19.536 s] > [INFO] Apache Hadoop YARN Server .......................... SUCCESS [ > 0.080 s] > [INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ > 11.128 s] > [INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ > 16.997 s] > [INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ > 1.623 s] > [INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ > 4.663 s] > [INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ > 2.247 s] > [INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ > 26.548 s] > [INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ > 2.960 s] > [INFO] Apache Hadoop YARN Client .......................... SUCCESS [ > 4.913 s] > [INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ > 1.732 s] > [INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ > 1.867 s] > [INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ > 0.123 s] > [INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [ > 3.153 s] > [INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [ > 3.503 s] > [INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [ > 0.088 s] > [INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SUCCESS [ > 4.038 s] > [INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [ > 3.494 s] > [INFO] Apache Hadoop YARN Router .......................... SUCCESS [ > 2.660 s] > [INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [ > 1.713 s] > [INFO] Apache Hadoop YARN Applications .................... SUCCESS [ > 0.091 s] > [INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ > 2.062 s] > [INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ > 1.544 s] > [INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ > 0.601 s] > [INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ > 6.887 s] > [INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ > 6.869 s] > [INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ > 3.005 s] > [INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ > 5.289 s] > [INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ > 4.543 s] > [INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ > 5.421 s] > [INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ > 1.516 s] > [INFO] Apache Hadoop YARN Services ........................ SUCCESS [ > 0.077 s] > [INFO] Apache Hadoop YARN Services Core ................... SUCCESS [ > 7.382 s] > [INFO] Apache Hadoop YARN Services API .................... SUCCESS [ > 2.310 s] > [INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [ > 0.086 s] > [INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [ > 24.634 s] > [INFO] Apache Hadoop YARN Application Catalog Docker Image SUCCESS [ > 0.183 s] > [INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [ > 0.128 s] > [INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [ > 1.844 s] > [INFO] Apache Hadoop YARN Site ............................ SUCCESS [ > 0.109 s] > [INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ > 1.188 s] > [INFO] Apache Hadoop YARN UI .............................. SUCCESS [ > 0.123 s] > [INFO] Apache Hadoop YARN CSI ............................. SUCCESS [ > 9.718 s] > [INFO] Apache Hadoop YARN Project ......................... SUCCESS [ > 18.364 s] > [INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ > 1.582 s] > [INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [ > 2.293 s] > [INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [ > 1.526 s] > [INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ > 2.003 s] > [INFO] Apache Hadoop MapReduce ............................ SUCCESS [ > 4.660 s] > [INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ > 1.782 s] > [INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ > 2.608 s] > [INFO] Apache Hadoop Client Aggregator .................... SUCCESS [ > 2.700 s] > [INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [ > 2.079 s] > [INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [ > 2.756 s] > [INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [ > 1.692 s] > [INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [ > 4.494 s] > [INFO] Apache Hadoop Dynamometer .......................... SUCCESS [ > 0.085 s] > [INFO] Apache Hadoop Archives ............................. SUCCESS [ > 1.052 s] > [INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ > 1.228 s] > [INFO] Apache Hadoop Rumen ................................ SUCCESS [ > 2.137 s] > [INFO] Apache Hadoop Gridmix .............................. SUCCESS [ > 1.913 s] > [INFO] Apache Hadoop Data Join ............................ SUCCESS [ > 1.181 s] > [INFO] Apache Hadoop Extras ............................... SUCCESS [ > 1.364 s] > [INFO] Apache Hadoop Pipes ................................ SUCCESS [ > 0.079 s] > [INFO] Apache Hadoop OpenStack support .................... SUCCESS [ > 1.598 s] > [INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ > 6.910 s] > [INFO] Apache Hadoop Kafka Library support ................ SUCCESS [ > 1.256 s] > [INFO] Apache Hadoop Azure support ........................ SUCCESS [ > 4.236 s] > [INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ > 1.051 s] > [INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ > 2.416 s] > [INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ > 2.483 s] > [INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ > 1.284 s] > [INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [ > 1.575 s] > [INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ > 5.312 s] > [INFO] Apache Hadoop Tools ................................ SUCCESS [ > 0.074 s] > [INFO] Apache Hadoop Client API ........................... SUCCESS [03:17 > min] > [INFO] Apache Hadoop Client Runtime ....................... SUCCESS [02:59 > min] > [INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [ > 1.076 s] > *[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [10:32 > min]* > [INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [ > 2.611 s] > [INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [ > 0.533 s] > [INFO] Apache Hadoop Distribution ......................... SUCCESS [02:55 > min] > [INFO] Apache Hadoop Client Modules ....................... SUCCESS [ > 0.168 s] > [INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [ > 2.697 s] > [INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ > 2.792 s] > > it used to be that -DskipShade would save the shading, but last time I > tried to skip the shading something failed. > > 1. has anyone else tried a full dist build recently -and how long did it > take? > 2. could they skip the shading? >
