[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777131#comment-16777131 ] Anu Engineer commented on HDDS-891: --- bq. No one is willing to do the work, apparently. Of course, it doesn't help that ozone and submarine build integrations are full of bad decisions. Would you be willing to give us some guidance, I am willing to work on this. But this is not something that I know well. So I am open to a design sketch and if you can provide that I am willing to go fix this. Thanks > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777128#comment-16777128 ] Allen Wittenauer commented on HDDS-891: --- bq. Unfortunately yetus doesn't work very well for the patches for ozone/submarine. No one is willing to do the work, apparently. Of course, it doesn't help that ozone and submarine build integrations are full of bad decisions. > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[
https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16776708#comment-16776708
]
Elek, Marton commented on HDDS-891:
---
Thanks [~aw] the answer. I created HADOOP-16146 to fix the issue in
start-build-env.sh
{quote} nobody should be playing with changes to stuff there without yetus
doing a build of hadoop.
{quote}
I couldn't agree more. Unfortunately yetus doesn't work very well for the
patches for ozone/submarine. The whole question for me was that how can it be
even more safe.
> Create customized yetus personality for ozone
> -
>
> Key: HDDS-891
> URL: https://issues.apache.org/jira/browse/HDDS-891
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
>
> Ozone pre commit builds (such as
> https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop
> personality from the yetus personality.
> Yetus personalities are bash scripts which contain personalization for
> specific builds.
> The hadoop personality tries to identify which project should be built and
> use partial build to build only the required subprojects because the full
> build is very time consuming.
> But in Ozone:
> 1.) The build + unit tests are very fast
> 2.) We don't need all the checks (for example the hadoop specific shading
> test)
> 3.) We prefer to do a full build and full unit test for hadoop-ozone and
> hadoop-hdds subrojects (for example the hadoop-ozone integration test always
> should be executed as it contains many generic unit test)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774087#comment-16774087 ] Steve Loughran commented on HDDS-891: - bq. (e.g., the last two changes to /pom.xml came from HDDS!). nobody should be playing with changes to stuff there without yetus doing a build of hadoop. > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773432#comment-16773432 ] Anu Engineer commented on HDDS-891: --- bq. well, [~anu] should never have +1'd without a precommit run and it should definitely have not been committed without it. Thanks for pointing it out, How can I fix these? Can you please share the command to run the shell checks locally, so I can do the best to remedy the situation? > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773255#comment-16773255 ] Allen Wittenauer commented on HDDS-891: --- 1. bq. If I understood well, you suggest to double quote the $DOCKER_INTERACTIVE_RUN variable in the docker run line. It's not even that. To do what you are wanting to do should really just be an extra flag to set the extra options rather than a full-blown 'set it from the outside' variable. bq. But please let me know if I am wrong. Let me do one better and give you an example. DOCKER_INTERACTIVE_RUN opens the door for users to set command line options to docker. Most notably, -c and -v and a few others that share one particular characteristic: they reference the file system. As soon as shell code hits the file system, it is no longer safe to assume space delimited options. In other words, -c /My Cool Filesystem/Docker Files/config.json or -v /c_drive/Program Files/Data:/data may be something a user wants to do, but the script now breaks because of the IFS assumptions. This bug is exactly why shellcheck is correctly flagging it as busted code. 2. bq. running docker based acceptance tests, While it is not well tested, it should be doable with 0.9.0+ with --dockerind mode. If external volumes are required to be mounted, things might get wonky though. Just be aware that users and other ASF build server patrons get annoyed when jobs take too long during peak hours. precommit response should be quick checks with better, post-commit checks happening at full build time. If they can't be triggered from maven test, then a custom test needs to be defined, either in the personality (recommended) or in the --user-plugins directory (not recommended, mainly because people will forget to set this option when they run test-patch interactively). bq. run ALL the hdds/ozone unit tests all the time, not just for the changed projects See above. Full tests are run as part of the nightlies due to time constraints. bq. check ALL the findbugs/checkstyle issues not just the new ones For findbugs, that's the --findbugs-strict-precheck option which AFAIK most/all of the Hadoop jobs have enabled. It will fail the patch if there are pre-existing findbugs issues. Adding a similar option to checkstyle wouldn't be hard, but a reminder that this info is also presented in the nightlies. Also, if the source tree is already clean, then new checkstyle failures should technically be 'all' already. Experience has shown, though, that users tend to blow right past precheck failures and commit code anyway. [Hell, many PMC members ignore errors that their own patches generated, blaming the Jenkins nodes when it's pretty clear that their Java code has e.g., javadoc errors.] 3. bq. I am convinced to run the more strict tests in addition to the existing yetus tests. It sounds like everything you want is either already there or is fairly trivial to implement. bq. Please let me know If I can do something to get Yetus results for the PR-s. I think [~ste...@apache.org] just needs to edit the user/pw for the hadoop-multibranch job credentials and HADOOP-16035 committed. I do practically zero Java these days, so it may not be 100% and probably needs a few more tweaks after it is implemented. (A definite flaw with Jenkins' multibranch pipelines.) > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[
https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772771#comment-16772771
]
Elek, Marton commented on HDDS-891:
---
Thanks the answer [~aw]
Sorry for the confusion. I think there are two questions which are mixed here.
# I would like to fix HDDS-146. It was not clear for me what is the wrong. If
I understood well, you suggest to double quote the $DOCKER_INTERACTIVE_RUN
variable in the docker run line. I think it's a false positive report as
DOCKER_INTERACTIVE_RUN should be either an empty string or unset (which will be
replaced by "-i -t"). I think it shouldn't be quoted because in that case
instead of -i -t we would add just one argument ('-i -t'). But please let me
know if I am wrong. (I can add a check if it initial value anything other than
an empty string.)
# "_Cutting back on options just because you think they don't apply to Ozone
is almost entirely incorrect."_ Please not that I am totally on the _opposite_
side. I would like to add MORE tests and MORE strict tests _in addition_ to the
existing tests. (Especially: running docker based acceptance tests, run ALL the
hdds/ozone unit tests all the time, not just for the changed projects, check
ALL the findbugs/checkstyle issues not just the new ones).
# I am convinced to run the more strict tests in addition to the existing
yetus tests. Please let me know If I can do something to get Yetus results for
the PR-s.
> Create customized yetus personality for ozone
> -
>
> Key: HDDS-891
> URL: https://issues.apache.org/jira/browse/HDDS-891
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
>
> Ozone pre commit builds (such as
> https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop
> personality from the yetus personality.
> Yetus personalities are bash scripts which contain personalization for
> specific builds.
> The hadoop personality tries to identify which project should be built and
> use partial build to build only the required subprojects because the full
> build is very time consuming.
> But in Ozone:
> 1.) The build + unit tests are very fast
> 2.) We don't need all the checks (for example the hadoop specific shading
> test)
> 3.) We prefer to do a full build and full unit test for hadoop-ozone and
> hadoop-hdds subrojects (for example the hadoop-ozone integration test always
> should be executed as it contains many generic unit test)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770158#comment-16770158 ] Allen Wittenauer commented on HDDS-891: --- I already answered your Yetus question: if ozone needs special handling for its submodules, it needs to go into the hadoop personality. Cutting back on options just because you think they don't apply to Ozone is almost entirely incorrect. It's still part of Hadoop and it's still very possible for people to modify core Hadoop through it. e.g., HDDS-1115, filed by you, does exactly that. The project doesn't have enough people working on it anymore to have wildly different build characteristics. The fewer variances between Hadoop subprojects the easier it is for the handful of people still working on it. HADOOP-16035 really drives this point home: there is, without a significant amount of hacks or--as the PR jobs you've already got setup demonstrate--massive confusion, only one precommit for github PRs for all source code sitting in the hadoop tree. Never mind that users running test-patch locally aren't going to know to use some oddball personality file that is off to the side. As to the various bugs introduced by HDDS-146... well, [~anu] should never have +1'd without a precommit run and it should definitely have not been committed without it. The shellcheck errors are through the roof. If it had been run or Ozone was paying attention to the nightly qbt runs, the errors are all laid out there. By far, the most problematic code is the use of unquoted variables for command line arguments. TBF, it's a common pitfall that a lot of inexperienced shell programmers hit, but given that we have tools in our build system to specifically point these types of errors out, there's little excuse for _new_ code to have this problem. [In hindsight, I should have incompatibility broken HADOOP_OPTS and friends in 3.x but it is what it is. It's been a bug in Hadoop since Doug copied the original code from Tomcat or Maven or whatever. Meanwhile, some users are pretty much required to do a lot of gymnastics to make the system work because of that long long long standing bug.] > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770078#comment-16770078 ] Elek, Marton commented on HDDS-891: --- I did it *before* your -1: I tried to create a personality and tested with Precommit-HDDS-Admin-nextgen and Precommit-HDDS-Admin how does it work: [https://gist.github.com/elek/315f251b71bfb8d5f66e99eafbca7808/revisions] --> Created on 13th of October This issue is created ~1.5 month later to use similar personality. The jenkins job is changed at the creation of this Jira (before your -1) [https://builds.apache.org/job/PreCommit-HDDS-Build/jobConfigHistory/showDiffFiles?timestamp1=2018-10-07_23-04-54=2018-12-03_09-06-15] Now I am really stuck, and I don't know how should I go forward with Yetus. And I would really appreciate any help (and answers)! (ps: you are right, I didn't modify the job since then, because I didn't know what should I do, I tried to continue the discussion. And thanks the reminder, I removed the test personality from the job configuration.) Can you please answer my questions and give me instructions how can we achieve the goals which are defined above? Thanks a lot. > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[
https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770031#comment-16770031
]
Allen Wittenauer commented on HDDS-891:
---
{code}
https://gist.githubusercontent.com/elek/315f251b71bfb8d5f66e99eafbca7808/raw/a184384a5e13c345362fd15661584e5984886f51/ozone.sh
{code}
Why did you make this change despite the -1?
> Create customized yetus personality for ozone
> -
>
> Key: HDDS-891
> URL: https://issues.apache.org/jira/browse/HDDS-891
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
>
> Ozone pre commit builds (such as
> https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop
> personality from the yetus personality.
> Yetus personalities are bash scripts which contain personalization for
> specific builds.
> The hadoop personality tries to identify which project should be built and
> use partial build to build only the required subprojects because the full
> build is very time consuming.
> But in Ozone:
> 1.) The build + unit tests are very fast
> 2.) We don't need all the checks (for example the hadoop specific shading
> test)
> 3.) We prefer to do a full build and full unit test for hadoop-ozone and
> hadoop-hdds subrojects (for example the hadoop-ozone integration test always
> should be executed as it contains many generic unit test)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756039#comment-16756039 ] Elek, Marton commented on HDDS-891: --- bq. Yup: HDDS-146, for example, changed start-build-env.sh (and introduced a bug) [~aw] Please let me know if you have information about the mentioned bug. > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744070#comment-16744070 ] Elek, Marton commented on HDDS-891: --- Can you please [~aw] help me to understand the mentioned bug? I am eager to fix it. > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[
https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16735644#comment-16735644
]
Elek, Marton commented on HDDS-891:
---
{quote}Yup: HDDS-146, for example, changed start-build-env.sh (and introduced a
bug). So clearly there is still some dependence despite everything said
above{quote}
BTW: I would like to fix the bug there. Would you be so kind [~aw] to give me a
hint where is it reported or create a jira if it's not yet reported? Thanks a
lot.
ps: HDDS-146 was merged before an additional check which is suggested in the
comments. So it won't be happened any more with the proposed solution.
> Create customized yetus personality for ozone
> -
>
> Key: HDDS-891
> URL: https://issues.apache.org/jira/browse/HDDS-891
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
>
> Ozone pre commit builds (such as
> https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop
> personality from the yetus personality.
> Yetus personalities are bash scripts which contain personalization for
> specific builds.
> The hadoop personality tries to identify which project should be built and
> use partial build to build only the required subprojects because the full
> build is very time consuming.
> But in Ozone:
> 1.) The build + unit tests are very fast
> 2.) We don't need all the checks (for example the hadoop specific shading
> test)
> 3.) We prefer to do a full build and full unit test for hadoop-ozone and
> hadoop-hdds subrojects (for example the hadoop-ozone integration test always
> should be executed as it contains many generic unit test)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714382#comment-16714382 ] Elek, Marton commented on HDDS-891: --- bq. and for reasons which have yet to be justified as to why that is desirable. Sorry if it was not clear enough from my last paragraph, my fault. The reason is the simplicity and maintainability. I believe that one project could be built with two separated personalities: 1) if it's more easy to maintain 2) there are two separated build paths. I agree with you: we need to rethink the whole build process when we will deliver ozone together with the main hadoop release. > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714317#comment-16714317 ] Allen Wittenauer commented on HDDS-891: --- bq. YOr is the problem to check a modification which is filed under HADOOP but modifies something under hadoop-ozone/hadoop-hdds? I don't think it it's handled right now (so we are as good as now), and I didn't see any example for that. We've already been seeing this fail in the nightly qbt since ozone got committed. Whether we see changes happening anywhere else or not is irrelevant. bq. Can't see any problem here. A full (hadoop + ozone) checkstyle should execute exactly the same checkstyle rules which are checked by the ozone personality. It currently does not. bq. For me using hadoop + ozone personalities seems to be a more clean separation. Ozone is part of Hadoop. The whole point of making it that was, from what I can tell, to get co-bundled at some point in the future. Making a separate personality is exactly the opposite direction and for reasons which have yet to be justified as to why that is desirable. My -1 remains. > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714112#comment-16714112 ] Elek, Marton commented on HDDS-891: --- bq. a) It's still possible to modify these components from the other JIRA categories. Yes, but for other jira categories other pre-commit build will be executed. We can check the hdds/ozone changes with ozone personality and other changes (HADOOP/HDDS/MAPREDUCE) can be checked with the hadoop personalities. I think we agreed that we shouldn't mix the two kind of changes in one patch. Or is the problem to check a modification which is filed under HADOOP but modifies something under hadoop-ozone/hadoop-hdds? I don't think it it's handled right now (so we are as good as now), and I didn't see any example for that. Worst case we can also improve the hadoop personality to fail in case of a hadoop-ozone/hadoop-hdds changes are included. bq. b) This part of the project still needs the capability to modify other modules (e.g., hadoop-dist) Not sure. Ozone is packaged by hadoop-ozone/dist maven project and not hadoop-dist project. If we need to modify something to modify outside the hadoop-ozone/hadoop-hdds projects I propose to do this in a HADOOP jira which are checked with the hadoop personality. bq. c) qbt runs over the entire source repository Can't see any problem here. A full (hadoop + ozone) checkstyle should execute exactly the same checkstyle rules which are checked by the ozone personality. BTW, according to the original plan hdds profile should be turned off by default, so qbt shouldb't check the ozone subproject. I am not sure which one is better I am open to both: 1) check everything in one qbt, or use two qbts: one for hadoop one for ozone >From technical point of view I think all the requirements can be implemented >both by only one (hadoop) or two (hadoop + ozone) personalities. For me using >hadoop + ozone personalities seems to be a more clean separation. To put >everything in the hadoop personality makes it more complex and we need a lot >of additional condition. Dont' we? (And there is a bigger chance to break something on the pure hadoop side with an ozone modification in the hadoop personality). > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714106#comment-16714106 ] Allen Wittenauer commented on HDDS-891: --- bq. Why is it not enough? Because: a) It's still possible to modify these components from the other JIRA categories. b) This part of the project still needs the capability to modify other modules (e.g., hadoop-dist) c) qbt runs over the entire source repository d) It's incredibly short-sighted. I'm sure I'm forgetting things, but it doesn't really matter. Fundamentally, this stuff is part of the Hadoop source. > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711135#comment-16711135 ] Elek, Marton commented on HDDS-891: --- bq. There is no way to guarantee that further changes won't leak outside these two modules I think there is a way: I propose to create an additional check and fail the precommit build if anything has been changed outside the hdds and ozone projects. Why is it not enough? > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710786#comment-16710786 ] Allen Wittenauer commented on HDDS-891: --- bq. Please note that the hadoop-ozone and hadoop-hdds in the current trunk doesn't depend on the in-tree hadoop-hdfs/common projects. It depends on the hadoop-3.2-SNAPSHOT as of now and we would like to switch to a stable release as soon as possible. That won't always be the case unless Ozone becomes its own project. There's no value in creating technical debt here. bq. Technically it's possible Yup: HDDS-146, for example, changed start-build-env.sh (and introduced a bug). So clearly there is still some dependence despite everything said above. bq. b): I like the idea and I tried to implement it. Would you be so kind to review the v2 patch of YETUS-631? YETUS-631 isn't an implementation of b at all. [and, FWIW, I'm going to reject that patch. I've got a better fix as part of YETUS-723.] bq. I checked the last two commits: If I understood well there was an additional property in the root pom.xml for ozone version (low risk) and with the last commit it was removed, so the parent pom.xml shouldn't be modified any more. Irrelevant. A change is a change. There is no way to guarantee that further changes won't leak outside these two modules short of not having any other code in the branch/repo. My binding vote remains -1. > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710480#comment-16710480 ] Elek, Marton commented on HDDS-891: --- Thanks the feedback [~aw]. Please note that the hadoop-ozone and hadoop-hdds in the current trunk doesn't depend on the in-tree hadoop-hdfs/common projects. It depends on the hadoop-3.2-SNAPSHOT as of now and we would like to switch to a stable release as soon as possible. Therefore it's not possible to modify both hdds/ozone AND hadoop in the same patch. Technically it's possible, but ozone/hdds code won't see the new changes in hdfs/common projects as they are 3.3-SNAPSHOT. We can add additional check that HDDS issues to ensure that the patches modify only the ozone/hdds subprojects and not the core hadoop/hdfs/common projects. With this modification I think it's safe to do a full ozone/hdds only build for hdds precommit hook. What do you think about it? about b): I like the idea and I tried to implement it. Would you be so kind to review the v2 patch of YETUS-631? > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710489#comment-16710489 ] Elek, Marton commented on HDDS-891: --- bq. (e.g., the last two changes to /pom.xml came from HDDS!). I checked the last two commits: If I understood well there was an additional property in the root pom.xml for ozone version (low risk) and with the last commit it was removed, so the parent pom.xml shouldn't be modified any more. > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710384#comment-16710384 ] Allen Wittenauer commented on HDDS-891: --- -1 I'm going to stop you from shooting yourself in the foot here. bq. But in Ozone: That list may be true for patches that only modify hadoop-ozone or hadoop-hdds maven modules, but patches uploaded to the HDDS project sometimes hit more than just those modules (e.g., the last two changes to /pom.xml came from HDDS!). Plus the union between those two modules is the root of the tree which means it really is building everything. It's important to also remember that hadoop-ozone and hadoop-hdds are fair game to be modified in by other JIRA projects as well. Consider some other things: a) Re-arrange the source such that there aren't two children off of / such that modifying both modules won't trigger a full build. b) Modify the existing personality to slap -Phdds when the changed files list includes hadoop-ozone and hadoop-hdds. c) It should probably also trigger a native library build. (There's a tremendous amount of inconsistency with the test runs presently.) d) Modify the personality to skip shaded if the patch ONLY modifies hadoop-ozone/hadoop-hdds With the exception of the first, these are all pretty trivial to do and have a way higher chance of success. > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-891) Create customized yetus personality for ozone
[ https://issues.apache.org/jira/browse/HDDS-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16706844#comment-16706844 ] Elek, Marton commented on HDDS-891: --- I created a second precommit build: https://builds.apache.org/job/PreCommit-HDDS-Build-nextgen which uses a temporary personality from this gist: https://gist.githubusercontent.com/elek/315f251b71bfb8d5f66e99eafbca7808/raw/a184384a5e13c345362fd15661584e5984886f51/ozone.sh If it works well I will upload the personality as a patch and we can commit it to the hadoop-ozone/dev-support and use this in-tree personality. > Create customized yetus personality for ozone > - > > Key: HDDS-891 > URL: https://issues.apache.org/jira/browse/HDDS-891 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > Ozone pre commit builds (such as > https://builds.apache.org/job/PreCommit-HDDS-Build/) use the official hadoop > personality from the yetus personality. > Yetus personalities are bash scripts which contain personalization for > specific builds. > The hadoop personality tries to identify which project should be built and > use partial build to build only the required subprojects because the full > build is very time consuming. > But in Ozone: > 1.) The build + unit tests are very fast > 2.) We don't need all the checks (for example the hadoop specific shading > test) > 3.) We prefer to do a full build and full unit test for hadoop-ozone and > hadoop-hdds subrojects (for example the hadoop-ozone integration test always > should be executed as it contains many generic unit test) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
