[
https://issues.apache.org/jira/browse/HDDS-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16877161#comment-16877161
]
Eric Yang commented on HDDS-1734:
---------------------------------
[~anu]
{quote}Clean full builds are good; we should encourage more of them; instead of
bypassing them. We need to make our builds faster and try to avoid doing
complicated things in the build so that developers are encouraged to build
cleanly all the time.
{quote}
When doing a google search on github.com/apache, and look for
maven-assembly-plugin. BigData projects using maven-assembly-plugin are: spark,
hbase, flink, zookeeper, ranger, knox, hive, zeppelin, nifi, ambari, oozie,
tez, chukwa and many more. I think it covers all the projects that we are
working with. Hadoop was an exception for a long time because HADOOP-7642
steered the creation of tarball away from maven assembly.
However, it created inability to publish hadoop tarball to a nexus repository.
The goal is to ensure that we have a way to keep released Ozone tarball in
nexus repository. I recommend to give back user the power to make choice for
themselves.
[~elek]
{code:java}
cd hadoop-ozone/dist
mvn docker:build -Pdocker-build
{code}
The above command doesn't work with mvn clean docker:build -Pdocker-build.
Maven may not filter Dockerfile resource properly sometimes when clean goal can
not be performed. Hacks to prevent mvn clean from working is problematic at
best. I am more and more incline to make layout stitching script to use maven
assembly plugin to make mvn clean work.
{quote}I agree with Anu Engineer: It makes the build (and release) slower and
consumes more space. And I can't see any immediate technical benefit.
{quote}
By using mvn package, there is no disk space wasted or cause the build to be
slower. Please explain where this change consumes more space?
Performance benchmark for running mvn package -Pdist in dist project:
{code:java}
[INFO] --- exec-maven-plugin:1.3.1:exec (tar-ozone) @ hadoop-ozone-dist ---
$ tar -c -f ozone-0.5.0-SNAPSHOT.tar ozone-0.5.0-SNAPSHOT
$ gzip -f ozone-0.5.0-SNAPSHOT.tar
Ozone dist tar available at:
/home/eyang/test/hadoop/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT.tar.gz
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (dist-enforce) @
hadoop-ozone-dist ---
[INFO]
[INFO] --- maven-site-plugin:3.6:attach-descriptor (attach-descriptor) @
hadoop-ozone-dist ---
[INFO] No site descriptor found: nothing to attach.
[INFO]
[INFO] --- maven-dependency-plugin:3.0.2:build-classpath
(add-classpath-descriptor) @ hadoop-ozone-dist ---
[INFO] Skipped writing classpath file
'/home/eyang/test/hadoop/hadoop-ozone/dist/target/classpath'. No changes found.
[INFO]
[INFO] --- build-helper-maven-plugin:1.9:attach-artifact
(attach-classpath-artifact) @ hadoop-ozone-dist ---
[INFO]
[INFO] --- maven-jar-plugin:2.5:test-jar (default) @ hadoop-ozone-dist ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21.960 s
[INFO] Finished at: 2019-07-02T13:09:22-04:00
[INFO] Final Memory: 55M/758M
[INFO] ------------------------------------------------------------------------
{code}
{code:java}
[INFO] --- maven-assembly-plugin:3.1.1:single (tar-ozone) @ hadoop-ozone-dist
---
[INFO] Reading assembly descriptor: src/assembly/binary.xml
[INFO] Building tar:
/home/eyang/hadoop/hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT.tar.gz
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 22.209 s
[INFO] Finished at: 2019-07-02T13:07:27-04:00
[INFO] Final Memory: 57M/806M
[INFO] ------------------------------------------------------------------------
{code}
The time difference is marginal at best. Disk usage is identical:
With tar stitching:
{code}
[eyang@localhost dist]$ du -sh target/
731M target/
{code}
With maven-assembly-plugin:
{code}
[eyang@localhost dist]$ du -sh target/
731M target/
{code}
> Use maven assembly to create ozone tarball image
> ------------------------------------------------
>
> Key: HDDS-1734
> URL: https://issues.apache.org/jira/browse/HDDS-1734
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Eric Yang
> Assignee: Eric Yang
> Priority: Major
> Attachments: HDDS-1734.001.patch, HDDS-1734.002.patch,
> HDDS-1734.003.patch
>
>
> Ozone is using tar stitching to create ozone tarball. This prevents down
> stream project to use Ozone tarball as a dependency. It would be nice to
> create Ozone tarball with maven assembly plugin to have ability to cache
> ozone tarball in maven repository. This ability allows docker build to be a
> separate sub-module and referencing to Ozone tarball. This change can help
> docker development to be more agile without making a full project build.
> Test procedure:
> {code:java}
> mvn -f pom.ozone.xml clean install -DskipTests -DskipShade
> -Dmaven.javadoc.skip -Pdist{code}
> Expected result:
> This will install tarball into:
> {code:java}
> ~/.m2/repository/org/apache/hadoop/hadoop-ozone-dist/0.5.0-SNAPSHOT/hadoop-ozone-dist-0.5.0-SNAPSHOT.tar.gz{code}
> Test procedure 2:
> {code:java}
> mvn -f pom.ozone.xml clean package -DskipTests -DskipShade
> -Dmaven.javadoc.skip -Pdist{code}
>
> Expected result:
> hadoop/hadoop-ozone/dist/target directory contains
> ozone-0.5.0-SNAPSHOT.tar.gz file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
