[
https://issues.apache.org/jira/browse/HDDS-447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626157#comment-16626157
]
Hudson commented on HDDS-447:
-----------------------------
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15040 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15040/])
HDDS-447. Separate ozone-dist and hadoop-dist projects with real (bharat: rev
62f817d32e6304b4cc1f9b909e7b815316b3a021)
* (delete) hadoop-dist/src/main/compose/ozonescripts/start.sh
* (delete) hadoop-dist/src/main/compose/ozonescripts/.ssh/id_rsa
* (add) hadoop-ozone/dist/src/main/compose/ozonescripts/docker-config
* (add) hadoop-ozone/dist/src/main/smoketest/test.sh
* (add) hadoop-ozone/dist/src/main/compose/ozone-hdfs/docker-compose.yaml
* (add) hadoop-ozone/dist/src/main/smoketest/basic/ozone-shell.robot
* (delete) hadoop-ozone/ozone-manager/src/main/webapps/ozoneManager/main.html
* (delete) hadoop-hdds/server-scm/src/main/webapps/scm/scm-overview.html
* (edit) hadoop-hdds/tools/pom.xml
* (delete) dev-support/bin/ozone-dist-layout-stitching
* (delete)
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSQLCli.java
* (add) hadoop-ozone/dist/src/main/compose/ozoneperf/docker-config
* (delete) hadoop-dist/src/main/compose/ozoneperf/docker-config
* (delete) hadoop-dist/src/main/ozone/README.txt
* (delete)
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSQLCli.java
* (add) hadoop-ozone/dist/src/main/smoketest/ozonefs/ozonefs.robot
* (add) hadoop-ozone/dist/src/main/compose/ozonescripts/.ssh/environment
* (delete) hadoop-dist/src/main/compose/ozoneperf/docker-compose.yaml
* (delete) hadoop-dist/src/main/compose/ozonescripts/README.md
* (add) hadoop-ozone/dist/src/main/compose/ozonescripts/stop.sh
* (add) hadoop-ozone/dist/src/main/compose/ozonescripts/.ssh/id_rsa.pub
* (add) hadoop-ozone/dist/src/main/compose/ozoneperf/init.sh
* (edit) hadoop-ozone/objectstore-service/pom.xml
* (delete) hadoop-ozone/ozone-manager/src/main/webapps/ozoneManager/index.html
* (add) hadoop-hdds/server-scm/src/main/resources/webapps/scm/main.html
* (delete) hadoop-dist/src/main/compose/ozonescripts/docker-config
* (add) hadoop-ozone/dist/src/main/compose/ozonefs/docker-config
* (delete) hadoop-dist/src/main/compose/ozonescripts/docker-compose.yaml
* (delete) hadoop-hdds/server-scm/src/main/webapps/scm/scm.js
* (edit) hadoop-ozone/pom.xml
* (add) hadoop-ozone/dist/src/main/compose/ozonescripts/README.md
* (delete)
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestDataValidate.java
* (delete) hadoop-dist/src/main/compose/ozonescripts/.ssh/authorized_keys
* (add) hadoop-ozone/dist/src/main/compose/ozone/.env
* (add) hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml
* (add) hadoop-ozone/dist/src/main/compose/ozonescripts/.ssh/config
* (delete) hadoop-dist/src/main/compose/ozoneperf/docker-compose-freon.yaml
* (add) hadoop-ozone/dist/src/main/smoketest/basic/basic.robot
* (delete)
hadoop-ozone/ozone-manager/src/main/webapps/ozoneManager/ozoneManager.js
* (add)
hadoop-ozone/ozone-manager/src/main/resources/webapps/ozoneManager/ozoneManager.js
* (edit) hadoop-ozone/client/pom.xml
* (delete) hadoop-dist/src/main/compose/ozonescripts/ps.sh
* (delete) hadoop-dist/src/main/compose/ozone/.env
* (edit) hadoop-ozone/common/pom.xml
* (add) hadoop-ozone/dist/src/main/smoketest/README.md
* (edit) hadoop-ozone/integration-test/pom.xml
* (delete) hadoop-dist/src/main/smoketest/basic/basic.robot
* (add) hadoop-ozone/dist/src/main/compose/ozoneperf/docker-compose-freon.yaml
* (add) hadoop-ozone/dist/src/main/compose/ozonescripts/ps.sh
* (delete) dev-support/bin/ozone-dist-tar-stitching
* (delete) hadoop-dist/src/main/compose/ozoneperf/init.sh
* (delete) hadoop-hdds/server-scm/src/main/webapps/scm/index.html
* (add) hadoop-ozone/dist/src/main/compose/ozone-hdfs/.env
* (add)
hadoop-ozone/ozone-manager/src/main/resources/webapps/ozoneManager/om-metrics.html
* (add) hadoop-hdds/server-scm/src/main/resources/webapps/scm/scm.js
* (add) hadoop-ozone/dist/src/main/compose/ozonescripts/.ssh/id_rsa
* (add) hadoop-ozone/dist/src/main/compose/ozonescripts/start.sh
* (add) hadoop-ozone/dist/src/main/compose/ozone/docker-config
* (delete) hadoop-dist/src/main/compose/ozonescripts/.ssh/id_rsa.pub
* (delete) hadoop-dist/src/main/compose/ozonefs/docker-config
* (add) hadoop-ozone/dist/src/main/compose/ozoneperf/docker-compose.yaml
* (delete) hadoop-dist/src/main/compose/ozonescripts/.ssh/config
* (edit) hadoop-ozone/docs/pom.xml
* (edit) hadoop-hdds/client/pom.xml
* (edit) hadoop-ozone/tools/pom.xml
* (add) hadoop-ozone/dist/src/main/compose/ozonescripts/Dockerfile
* (add)
hadoop-ozone/ozone-manager/src/main/resources/webapps/ozoneManager/main.html
* (add) hadoop-ozone/dist/dev-support/bin/dist-tar-stitching
* (add) hadoop-ozone/dist/pom.xml
* (edit) hadoop-ozone/ozone-manager/pom.xml
* (add)
hadoop-ozone/ozone-manager/src/main/resources/webapps/ozoneManager/index.html
* (edit) hadoop-hdds/common/pom.xml
* (add) hadoop-hdds/server-scm/src/main/resources/webapps/scm/scm-overview.html
* (delete) hadoop-dist/src/main/compose/ozonescripts/stop.sh
* (delete) hadoop-dist/src/main/smoketest/basic/ozone-shell.robot
* (add) hadoop-ozone/dist/src/main/smoketest/commonlib.robot
* (delete) hadoop-dist/src/main/smoketest/test.sh
* (delete) hadoop-ozone/ozone-manager/src/main/webapps/ozoneManager/main.css
* (add) hadoop-ozone/dist/src/main/compose/ozoneperf/compose-all.sh
* (delete) hadoop-dist/src/main/compose/ozone/docker-config
* (edit) hadoop-ozone/common/src/main/bin/ozone
* (add) hadoop-ozone/dist/src/main/ozone/README.txt
* (edit) hadoop-hdds/server-scm/pom.xml
* (delete) hadoop-dist/src/main/compose/ozone-hdfs/docker-compose.yaml
* (add)
hadoop-ozone/ozone-manager/src/main/resources/webapps/ozoneManager/main.css
* (edit) hadoop-hdds/container-service/pom.xml
* (add) hadoop-ozone/datanode/pom.xml
* (delete) hadoop-dist/src/main/compose/ozoneperf/README.md
* (delete) hadoop-hdds/server-scm/src/main/webapps/scm/main.html
* (add) hadoop-ozone/dist/src/main/compose/ozonescripts/docker-compose.yaml
* (add)
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/freon/TestRandomKeyGenerator.java
* (delete) hadoop-dist/src/main/compose/ozoneperf/.env
* (delete) hadoop-dist/src/main/compose/ozone-hdfs/.env
* (delete) hadoop-dist/src/main/smoketest/README.md
* (edit) hadoop-dist/pom.xml
* (add)
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/freon/TestDataValidate.java
* (add) hadoop-ozone/dist/src/main/compose/ozoneperf/README.md
* (delete) hadoop-dist/src/main/compose/ozoneperf/compose-all.sh
* (edit) hadoop-project/pom.xml
* (edit) hadoop-hdds/pom.xml
* (add) hadoop-ozone/dist/src/main/compose/ozoneperf/prometheus.yml
* (delete)
hadoop-ozone/ozone-manager/src/main/webapps/ozoneManager/om-metrics.html
* (add) hadoop-ozone/dist/dev-support/bin/dist-layout-stitching
* (delete) hadoop-dist/src/main/compose/ozonescripts/.ssh/environment
* (edit) hadoop-ozone/common/src/main/shellprofile.d/hadoop-ozone.sh
* (delete) hadoop-dist/src/main/compose/ozone-hdfs/docker-config
* (add)
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/om/TestOmSQLCli.java
* (add) hadoop-ozone/dist/src/main/compose/ozoneperf/.env
* (add) hadoop-ozone/dist/src/main/compose/ozone-hdfs/docker-config
* (delete) hadoop-dist/src/main/smoketest/commonlib.robot
* (delete) hadoop-dist/src/main/compose/ozonefs/docker-compose.yaml
* (edit) hadoop-hdds/framework/pom.xml
* (delete) hadoop-dist/src/main/compose/README.md
* (delete)
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestRandomKeyGenerator.java
* (delete) hadoop-dist/src/main/compose/ozone/docker-compose.yaml
* (add) hadoop-ozone/dist/src/main/compose/ozonescripts/.ssh/authorized_keys
* (add) hadoop-ozone/dist/src/main/compose/README.md
* (delete) hadoop-dist/src/main/compose/ozoneperf/prometheus.yml
* (add) hadoop-ozone/dist/src/main/compose/ozonefs/docker-compose.yaml
* (delete) hadoop-dist/src/main/compose/ozonescripts/Dockerfile
* (add) hadoop-hdds/server-scm/src/main/resources/webapps/scm/index.html
* (delete) hadoop-dist/src/main/smoketest/ozonefs/ozonefs.robot
* (edit) hadoop-ozone/ozonefs/pom.xml
* (add)
hadoop-ozone/tools/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSQLCli.java
> Separate ozone-dist and hadoop-dist projects with real classpath separation
> ---------------------------------------------------------------------------
>
> Key: HDDS-447
> URL: https://issues.apache.org/jira/browse/HDDS-447
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Reporter: Elek, Marton
> Assignee: Elek, Marton
> Priority: Major
> Fix For: 0.3.0
>
> Attachments: HDDS-447-ozone-0.2.001.patch, HDDS-447.003.patch,
> HDDS-447.004.patch, HDDS-447.005.patch, HDDS-447.006.patch,
> HDDS-447.007.patch, HDDS-447.008.patch
>
>
> Currently we have the same hadoop-dist project to create both the ozone and
> and the hadoop distribution.
> To decouple ozone and hadoop build it would be great to create two different
> dist project.
> The hadoop-dist should be cloned to hadoop-ozone/dist and from
> hadoop-dist/pom.xml we can remove the hdds/ozone related items and from
> hadoop-ozone/dist/pom.xml we can remove the core hadoop related part.
> An other issue with the current distribution schema is the lack of real
> classpath separation.
> The current hadoop distribution model is defined in the hadoop-project-dist
> which is parent of all the component projects and the output of the
> distribution generation will be copied by the dist-layout-stitching. There is
> no easy way to use command specific classpath as the classpath is defined in
> component level (hdfs/yarn/mapreduce).
> With this approach we will have a lot of unnecessary dependencies on the
> classpath (which were not on the classpath at the time of the unit tests) and
> it's not possible (as an example) use different type of jaxrs stack for
> different services (s3gateway vs scm).
> As a simplified but more effective approach I propose to use the following
> method:
> 1. don't use hadoop-project-dist for ozone projects any more
> 2. During the build generate a classpath descriptor (with the
> dependency:build-classpath maven plugin/goal) for all the projects
> 3. During the distribution copy all the required dependencies (with
> dependency:copy maven plugin/goal) to a lib folder (share/ozone/lib)
> 4. During the distribution copy all the classpath descriptors to the
> classpath folder (share/ozone/classpath)
> 5. Put only the required jar files to the classpath with reading the
> classpath descriptor
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
