[
https://issues.apache.org/jira/browse/HDDS-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16738295#comment-16738295
]
Elek, Marton commented on HDDS-965:
-----------------------------------
Thanks the comments [~busbey].
Personally I am always +1 to a discussion. Especially as I don't think we have
the final answer to the question how Ozone should be built.
It was voted by the community to keep the source in the same tree but use
different release lifecycle (which means slightly different build setup). But
until Ozone there was no such subproject in Hadoop, we have no clear pattern
which can be followed.
During the previous Ozone releases we tried multiple approaches and the process
was continuously improved.
Some examples:
Originally we used the hadoop-dist project but after a while we separated
hadoop-dist and hadoop-ozone/dist (HDDS-447)
Originally we used the some classpath structure as hadoop (which includes test
jars + yarn jars on the classpath of the namenode), but after a while we
switched to use more specific classpath definition (same HDDS-447).
Originally we used the in-tree hadoop version, but now it's (almost) possible
to compile ozone with multiple different hadoop version.
I also agree that the build of hadoop subprojects should be as unified as
possible. But Ozone is an alpha software, some good new things can be tried
before moving them to the hadoop source.
Personally I think that the classpath separation of HDDS-447 should be used by
hdfs/yarn as well. But my feeling is that it's more risky to do such a
fundamental change in hadoop side. I would prefer to try it out with ozone
first and if works well we can discuss about the adoption for all the hadoop
components.
Similar for the testing. In Ozone we started to use docker-compose based pseudo
cluster definitions and scripted smoketest/acceptance test suite based on robot
framework. I think it's extremely useful not just to test Ozone as a user but
to test new patches during the code review (or test the release package during
the vote). I have a plan to open a jira and start similar work for hdfs/yarn as
well (see ./compose and ./smoketest ini the ozone distribution tar files if you
are interested).
It was not mentioned exactly what should be discussed (I guess not the
checkstyle changes or the scripts), but let me repeat here: This patch is not
about removing yetus support or remove anything from the existing build
process. It provides additional checks which may or may not useful. (They are
more strict checks she code can be better with these checks)
I would also prefer to discuss about the goals regarding to the ozone builds.
Our current problems are:
* We would like to run the docker based acceptance tests for each patch (dnd
is required)
* We would like to run full unit test / checks for all the
hadoop-hdds/hadoop-ozone project for all the patch
* The unit tests from hadoop-ozone/integration-test should be executed for all
the patches
* Some of the existing checks (for example shaded client) can be ignore for
hdds/ozone only patches
* I agreed with AW that it would be better to strictly avoid patches which
modifies both ozone/hdds and other hadoop subprojects in the same time.
* Ozone doesn't require the build of Yarn/mapreduce components (handled with
-pl + -am maven flags as of now)
* And I would prefer to do it at least as fast as the local build (2-3 minutes
ozone build as of now) but keep all the required safety options.
I think some of these (all?) can be implemented in yetus, but I am not sure
how. I would really appreciate any help here.
> Ozone: checkstyle improvements and code quality scripts
> -------------------------------------------------------
>
> Key: HDDS-965
> URL: https://issues.apache.org/jira/browse/HDDS-965
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
> Reporter: Elek, Marton
> Assignee: Elek, Marton
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Experimental scripts to test github pr capabilities after the github url
> move. The provided scripts are easier to use locally and provides more
> strict/focused checks then the existing pre-commit scripts. But this is not a
> replacements of the existing yetus build as it adds additional (more strict)
> checks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
