[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Release Note: * test-patch.sh now has new output that is different than the previous versions * test-patch.sh is now pluggable via the test-patch.d directory, with checkstyle and shellcheck tests included * JIRA comments now use much more markup to improve readability * test-patch.sh now supports either a file name, a URL, or a JIRA issue as input in developer mode * If part of the patch testing code is changed, test-patch.sh will now attempt to re-executing itself using the new version. * Some logic to try and reduce the amount of unnecessary tests. For example, patches that only modify markdown should not run the Java compilation tests. * Plugins for checkstyle, shellcheck, and whitespace now execute as necessary. * New test code for mvn site * A breakdown of the times needed to execute certain blocks as well as a total runtime is now reported to assist in fixing long running tests and optimize the entire process. * Several new options * --resetrepo will put test-patch.sh in destructive mode, similar to a normal Jenkins run * --testlist allows one to provide a comma delimited list of test subsystems to forcibly execute * --modulelist to provide a comma delimited list of module tests to execute in addition to the ones that are automatically detected * --offline mode to attempt to stop connecting to the Internet for certain operations * test-patch.sh now defaults to the POSIX equivalents on Solaris and Illumos-based operating systems * shelldocs.py may be used to generate test-patch.sh API information * FindBugs output is now listed on the JIRA comment * lots of general code cleanup, including attempts to remove any local state files to reduce potential race conditions * Some logic to determine if a patch is for a given major branch using several strategies as well as a particular git ref (using git+ref as part of the name). * Some logic to determine if a patch references a particular JIRA issue. * Unit tests are only flagged as necessary with native or Java code, since Hadoop has no framework in place yet for other types of unit tests. * test-patch now exits with a failure status if problems arise trying to do git checkouts. Previously the exit code was success. was: * test-patch.sh now has new output that is different than the previous versions * test-patch.sh is now pluggable via the test-patch.d directory, with checkstyle and shellcheck tests included * JIRA comments now use much more markup to improve readability * test-patch.sh now supports either a file name, a URL, or a JIRA issue as input in developer mode * If part of the patch testing code is changed, test-patch.sh will now attempt to re-executing itself using the new version. * Some logic to try and reduce the amount of unnecessary tests. For example, patches that only modify markdown should not run the Java compilation tests. * Plugins for checkstyle, shellcheck, and whitespace now execute as necessary. * New test code for mvn site * A breakdown of the times needed to execute certain blocks as well as a total runtime is now reported to assist in fixing long running tests and optimize the entire process. * Several new options * --resetrepo will put test-patch.sh in destructive mode, similar to a normal Jenkins run * --testlist allows one to provide a comma delimited list of test subsystems to forcibly execute * --modulelist to provide a comma delimited list of module tests to execute in addition to the ones that are automatically detected * --offline mode to attempt to stop connecting to the Internet for certain operations * test-patch.sh now defaults to the POSIX equivalents on Solaris and Illumos-based operating systems * shelldocs.py may be used to generate test-patch.sh API information * FindBugs output is now listed on the JIRA comment * lots of general code cleanup, including attempts to remove any local state files to reduce potential race conditions * Some logic to determine if a patch is for a given major branch using several strategies as well as a particular git ref (using git+ref as part of the name). * Some logic to determine if a patch references a particular JIRA issue. * Unit tests are only flagged as necessary with native or Java code, since Hadoop has no framework in place yet for other types of unit tests. * test-patch now exits with a failure status if problems arise trying to do git checkouts. Previously the exit code was success. > rewrite test-patch.sh > - > > Key: HADOOP-11746 > URL: https://issues.apache.org/jira/browse/HADOOP-11746 > Project: Hadoop Common > Issue Type: Test > Components: build, test >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-11746:
--
Attachment: HADOOP-11746-21.patch
-21:
A one line change in the site test:
{code}
+ add_jira_table +1 site There were no new javadoc warning messages.
---
+ add_jira_table +1 site Site still builds.
{code}
...
[~cnauroth], does your +1 still stand? I'm pretty much ready to commit this.
Promise. lol
rewrite test-patch.sh
-
Key: HADOOP-11746
URL: https://issues.apache.org/jira/browse/HADOOP-11746
Project: Hadoop Common
Issue Type: Test
Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch,
HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch,
HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch,
HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch,
HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch,
HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch,
HADOOP-11746-18.patch, HADOOP-11746-19.patch, HADOOP-11746-20.patch,
HADOOP-11746-21.patch
This code is bad and you should feel bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Release Note: * test-patch.sh now has new output that is different than the previous versions * test-patch.sh is now pluggable via the test-patch.d directory, with checkstyle and shellcheck tests included * JIRA comments now use much more markup to improve readability * test-patch.sh now supports either a file name, a URL, or a JIRA issue as input in developer mode * If part of the patch testing code is changed, test-patch.sh will now attempt to re-executing itself using the new version. * Some logic to try and reduce the amount of unnecessary tests. For example, patches that only modify markdown should not run the Java compilation tests. * Plugins for checkstyle, shellcheck, and whitespace now execute as necessary. * New test code for mvn site * A breakdown of the times needed to execute certain blocks as well as a total runtime is now reported to assist in fixing long running tests and optimize the entire process. * Several new options * --resetrepo will put test-patch.sh in destructive mode, similar to a normal Jenkins run * --testlist allows one to provide a comma delimited list of test subsystems to forcibly execute * --modulelist to provide a comma delimited list of module tests to execute in addition to the ones that are automatically detected * --offline mode to attempt to stop connecting to the Internet for certain operations * test-patch.sh now defaults to the POSIX equivalents on Solaris and Illumos-based operating systems * shelldocs.py may be used to generate test-patch.sh API information * FindBugs output is now listed on the JIRA comment * lots of general code cleanup, including attempts to remove any local state files to reduce potential race conditions * Some logic to determine if a patch is for a given major branch using several strategies as well as a particular git ref (using git+ref as part of the name). * Some logic to determine if a patch references a particular JIRA issue. * Unit tests are only flagged as necessary with native or Java code, since Hadoop has no framework in place yet for other types of unit tests. * test-patch now exits with a failure status if problems arise trying to do git checkouts. Previously the exit code was success. was: * test-patch.sh now has new output that is different than the previous versions * test-patch.sh is now pluggable via the test-patch.d directory, with checkstyle and shellcheck tests included * JIRA comments now use much more markup to improve readability * test-patch.sh now supports either a file name, a URL, or a JIRA issue as input in developer mode * If part of the patch testing code is changed, test-patch.sh will now attempt to re-executing itself using the new version. * Some logic to try and reduce the amount of unnecessary tests. For example, patches that only modify markdown should not run the Java compilation tests. * Plugins for checkstyle, shellcheck, and whitespace now execute as necessary. * New test code for mvn site * A breakdown of the times needed to execute certain blocks as well as a total runtime is now reported to assist in fixing long running tests and optimize the entire process. rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch, HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch, HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-18.patch, HADOOP-11746-19.patch, HADOOP-11746-20.patch, HADOOP-11746-21.branch-2.patch, HADOOP-11746-21.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-21.branch-2.patch branch-2 version. rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch, HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch, HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-18.patch, HADOOP-11746-19.patch, HADOOP-11746-20.patch, HADOOP-11746-21.branch-2.patch, HADOOP-11746-21.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) This has been committed to trunk and branch-2. Thanks everyone for the assistance! rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Fix For: 2.8.0 Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch, HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch, HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-18.patch, HADOOP-11746-19.patch, HADOOP-11746-20.patch, HADOOP-11746-21.branch-2.patch, HADOOP-11746-21.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-20.patch Yeah I think it was probably a mistake. Here's -20 which changes all of those to return 1, thus failing the jenkins build. I guess we'll see what happens. rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch, HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch, HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-18.patch, HADOOP-11746-19.patch, HADOOP-11746-20.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-19.patch -19: * Merged in the fixes from HADOOP-11845, otherwise rat checks fail because mvn site generates a file that violates rat * shellcheck test reports version of shellcheck * shellcheck find command was very broken * upgraded my local copy of shellcheck, which now includes some new checks. fixed errors from those new checks (mainly local blah=subshell issues) * somewhere along the way, the value of the initial cwd got screwed up. * re-arranged the findbugs code a bit to give more realistic times * if for some reason openssl or base64 throws an error, just send them to /dev/null rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch, HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch, HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch, HADOOP-11746-18.patch, HADOOP-11746-19.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-17.patch After sleeping on it, I think it's better to be safe than sorry in case of problems. A few extra lines for extra paranoia... -17: * During re-exec mode, make sure jenkins reports the console in the switching message in case something goes wrong. * In reset repo mode, always force checkout to trunk after cleaning the repo area. This might not actually matter in the grand scheme of things, but if something goes haywire in the middle this should provide some level of safety to get us back to a clean slate. * Don't switch to any branches that match: branch-0, branch-1, branch-2.[0-7]. It should be noted that currently the code to find a branch won't properly detect minor or micro version branches anyway, so the branch-2 minor version lockout is just in case someone fixes that bug later. ;) It should be safe to commit this to branch-2 and have the system come back to trunk for 'real' patches. rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch, HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch, HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-11746:
--
Attachment: HADOOP-11746-18.patch
-18:
* add --modulelist and --testlist options to forcibly add modules and test
subsystems to a run. there is no way to delete, but that's probably a good
thing really.
* removed the branch restrictions since this apparently won't impact jenkins
* the check for new tests actually uses the /test/ file path against the list
of changed files in the check for new tests in unit test mode. This prevents
some false positives and should improve the count in certain/rare situations.
Prior to this, it just looked for (something)/test(something) in the actual
patch... which meant test-patch itself would be cleared. (Ironically,
test-patch.sh is now capable of testing itself, so ... yeah. test-ception.)
* in developer mode when --resetrepo is used, the user output reminds that the
code the user chose to run in a destructive mode. (--resetrepo is the default
in jenkins mode, obviously)
* Added code to chmod +x all directories in echo_and_redirect. This should
cover every maven run.
* Cleaned up the eclipse description.
* Checkstyle now uses the ${PROJECT_NAME} var instead of the hard coded hadoop
value, making the code more portable.
* Reworded some debug output for test subsystems when they are triggered
* verify_needed_test was missing documentation
bq. Adding audience / stability comments is a really nice touch.
Thanks. If you run shelldoc.py (after HADOOP-11845 is applied... ugh) against
the code, you'll get a markdown document of the function calls. We're currently
using this same functionality in trunk to document hadoop-functions.sh as the
'official' shell API for shell profiles.
rewrite test-patch.sh
-
Key: HADOOP-11746
URL: https://issues.apache.org/jira/browse/HADOOP-11746
Project: Hadoop Common
Issue Type: Test
Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch,
HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch,
HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch,
HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch,
HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch,
HADOOP-11746-15.patch, HADOOP-11746-16.patch, HADOOP-11746-17.patch,
HADOOP-11746-18.patch
This code is bad and you should feel bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-11746:
--
Attachment: HADOOP-11746-16.patch
-16:
* Fixed, etc through all of the [~cnauroth]'s review comments. Much thanks!
Some notes:
{{checkstyle.sh}}
bq. 2. Do you think it's worthwhile to move the Python code into its own .py
file?
I've been debating this myself. Two advantages of doing it this way:
* the whole plug-in is self contained.
* the dev-support patch detection is a lot easier
{{shellcheck.sh}}
bq. 1. {{shellcheck_private_findbash}}: This looks for files in bin and sbin.
Do we also need libexec, which is currently used in hadoop-kms and
hadoop-hdfs-httpfs?
Great catch. I also added shellprofile.d while I was there!
{{whitespace.sh}}
Ugh. Good point. That should be [[:blank:]] not [[:space:]]. Easy fix. This
also means this works in non-C locales a bit better.
{{testpatch.sh}}
bq. 2. {{determine_needed_tests}}: I believe this would not identify tests for
a patch that contained only changes in test resources (not Java test code).
Examples of this include testConf.xml and editsStored and editsStored.xml.
Relevant code:
{code}
elif [[ ${i} =~ \.c$
...
|| ${i} =~ src/test
...
]]; then
hadoop_debug tests/native: ${i}
add_test javac
add_test unit
{code}
It should. If *any* file has src/test in its path, it will trigger the javac
and unit tests. It's really not tests/native, so maybe that should get
renamed to something else I guess. (This is counter to tests/java which also
triggers javadoc tests)
bq. 4. {{output_to_console}}: Today I learned that your secret talent is ASCII
elephant art.
I don't know what you are talking about. Maybe your patch download was bad? ;)
bq. 5. There are shellcheck warnings on lines 1158, 1573, 1732, 1751, and 1995.
Fixed or disabled all of these except two, which are a) false positives and b)
extremely hard to escape because they are actually awk commands in a multiline
pipe.
bq. 6. The old test-patch.sh output always included a hyperlink to the patch
file that it ran. Can we please add that? I always found that helpful for
knowing exactly which patch it was reporting on, in case multiple revisions got
uploaded quickly.
It actually does this when the patch provided is from either a JIRA or a URL.
This is consistent with the old code.
FWIW, I've uploaded the same patch into HADOOP-11820 and ran it in jenkins
mode. You'll see the shellcheck #'s and the patch URL there as well. Check out
that execution time. :D :D
rewrite test-patch.sh
-
Key: HADOOP-11746
URL: https://issues.apache.org/jira/browse/HADOOP-11746
Project: Hadoop Common
Issue Type: Test
Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch,
HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch,
HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch,
HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch,
HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch,
HADOOP-11746-15.patch, HADOOP-11746-16.patch
This code is bad and you should feel bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Release Note: * test-patch.sh now has new output that is different than the previous versions * test-patch.sh is now pluggable via the test-patch.d directory, with checkstyle and shellcheck tests included * JIRA comments now use much more markup to improve readability * test-patch.sh now supports either a file name, a URL, or a JIRA issue as input in developer mode * If part of the patch testing code is changed, test-patch.sh will now attempt to re-executing itself using the new version. * Some logic to try and reduce the amount of unnecessary tests. For example, patches that only modify markdown should not run the Java compilation tests. * Plugins for checkstyle, shellcheck, and whitespace now execute as necessary. * New test code for mvn site * A breakdown of the times needed to execute certain blocks as well as a total runtime is now reported to assist in fixing long running tests and optimize the entire process. was: * test-patch.sh now has new output that is different than the previous versions * test-patch.sh is now pluggable via the test-patch.d directory, with checkstyle and shellcheck tests included * JIRA comments now use much more markup to improve readability * test-patch.sh now supports either a file name, a URL, or a JIRA issue as input in developer mode * If part of the patch testing code is changed, test-patch.sh will now attempt to re-executing itself using those new versions. * Some logic to try and reduce the amount of unnecessary tests. For example, patches that only modify markdown should not run the Java compilation tests. * Plugins for checkstyle, shellcheck, and whitespace now execute as necessary. * New test code for mvn site rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch, HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch, HADOOP-11746-15.patch, HADOOP-11746-16.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-11746: --- Hadoop Flags: Incompatible change,Reviewed (was: Incompatible change) +1 for patch v16. All of my feedback has been addressed either by code changes or explaining parts of the logic that I misunderstood. Since this impacts everyone's day-to-day work, can we please hold off committing until Friday, 4/17, just in case there are other opinions? I'll send notice now to the dev lists. I have just one more procedural question. I see this is flagged as an incompatible change. While that's technically true, I don't think compatibility concerns apply to our internal dev tooling. If it's flagged as incompatible, then it may cause confusion for Hadoop end users who think they need to do something in their deployments to react to this incompatibility. What do you think? rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch, HADOOP-11746-12.patch, HADOOP-11746-13.patch, HADOOP-11746-14.patch, HADOOP-11746-15.patch, HADOOP-11746-16.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-13.patch -13: * fix an issue where a git ref may be in the future vs. the copy on disk. We no longer do a pull --rebase, but a fetch + checkout --force in jenkins mode. * shellcheck support beefed up, including now failing on new errors, not if the total count was higher. Also smarter about picking up on patches that patch shell code that are not in the normal places * cosmetic fixes for the test table * renamed populate_unit_table to populate_test_table to better reflect what it is * findbugs moved into its own optional test instead of being tied to javac so that pom.xml, native code, etc, won't trigger a run. (This does mean that testing findbugs itself will get missed now... but that's a very rare occurrence...) rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch, HADOOP-11746-12.patch, HADOOP-11746-13.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-11746:
--
Attachment: HADOOP-11746-12.patch
-12:
* add --resetrepo to simulate jenkins git repo nuker in developer mode
* don't send a jira message and abort early if the URL doesn't end in .patch
* Allow git(8 chars) or git(41 chars) as branch names for patch testing
* make it explicit in the report output if we were in branch or git ref mode
* fix some issues with checkstyle
* when using --resetrepo mode, shellcheck would erroneously flag .orig and .rej
files from previous patches since those aren't cleared by git due to .gitignore
* fix a few shellcheck warnings I had missed
* findbugs now gives a summary report on the console/jira output rather than
forcing folks to look at the full findbugs output (altho that is still listed
too!)
* set the TIMER to be current time rather 0 to prevent incredibly bogus,
decades long runtimes in case of early aborts
* re-order the flags in the options parser
* fix an issue with the test heuristics so that external plugins aren't
prematurely ignored
Testing itself, it says:
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | reexec | 00m 00s | dev-support patch detected. |
| {color:blue}0{color} | pre-patch | 00m 00s | Pre-patch trunk compilation is
healthy. |
| {color:red}-1{color} | @author | 00m 00s | The patch appears to contain 13
@author tags which the Hadoop community has agreed to not allow in code
contributions. |
| {color:green}+1{color} | whitespace | 00m 00s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | release audit | 00m 09s | The applied patch does not
increase the total number of release audit warnings. |
| {color:green}+1{color} | shellcheck | 00m 02s | There were no new shellcheck
issues. |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | shellcheck |
| git revision | f4b3fc5 / trunk |
| Console output | /artifact/patchprocess/console |
This message was automatically generated.
rewrite test-patch.sh
-
Key: HADOOP-11746
URL: https://issues.apache.org/jira/browse/HADOOP-11746
Project: Hadoop Common
Issue Type: Test
Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch,
HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch,
HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch,
HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch,
HADOOP-11746-12.patch
This code is bad and you should feel bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Status: Patch Available (was: Open) rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-10.patch -10: * fix the rat error * add better heuristics to determine the minimal amount of long running tests to execute: significantly faster run times for certain classes of patches * add a site compilation test; doc patches actually get tested * fix a few bugs in the checkstyle plugin * fix a few bugs in the whitespace plugin rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Status: Open (was: Patch Available) rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-11.patch -11: * clean up the test selector API so that plugins could easily use it. so now checkstyle and (sometimes) shellcheck only kick in when necessary * added mvn install check to javac or javadoc test check. suspect it really only needs javadoc, but let's be conservative for now * fixed the regexes on the test selector * added cmakelists.txt as a trigger for the native code checks rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Status: Open (was: Patch Available) rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Release Note: * test-patch.sh now has new output that is different than the previous versions * test-patch.sh is now pluggable via the test-patch.d directory, with checkstyle and shellcheck tests included * JIRA comments now use much more markup to improve readability * test-patch.sh now supports either a file name, a URL, or a JIRA issue as input in developer mode * If part of the patch testing code is changed, test-patch.sh will now attempt to re-executing itself using those new versions. * Some logic to try and reduce the amount of unnecessary tests. For example, patches that only modify markdown should not run the Java compilation tests. * Plugins for checkstyle, shellcheck, and whitespace now execute as necessary. * New test code for mvn site was: * test-patch.sh now has new output that is different than the previous versions * test-patch.sh is now pluggable via the test-patch.d directory, with checkstyle and shellcheck tests included * JIRA comments now use much more markup to improve readability * test-patch.sh now supports either a file name, a URL, or a JIRA issue as input in developer mode * If part of the patch testing code is changed, test-patch.sh will now re-executing itself using those new versions. Status: Patch Available (was: Open) rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch, HADOOP-11746-10.patch, HADOOP-11746-11.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-08.patch -08: * borrow some code from the folks at HBase to help make the checkstyle check more useful * add an EOL whitespace check bz i'm tired of checking for it myself. :D * fix some extraneous output when no tests have failed rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-08.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Status: Open (was: Patch Available) rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-08.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Status: Patch Available (was: Open) rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-08.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-09.patch -09: * -08 was the wrong file. woops. rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-08.patch, HADOOP-11746-09.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: (was: HADOOP-11746-08.patch) rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch, HADOOP-11746-09.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Status: Patch Available (was: Open) rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-07.patch -07: * fixes the duplicate test bit that was missing in -06 rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Status: Open (was: Patch Available) rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch, HADOOP-11746-07.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-06.patch -06: * lots of fixes and cleanup, including some minor perf fixes and removing a few needless temp files * On Solaris, use POSIX not SVID binaries * If a patch hits test-patch or smart-apply-path, short-circuit some pre-patch logic, then use the new versions as part of testing the patch rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Release Note: * test-patch.sh now has new output that is different than the previous versions * test-patch.sh is now pluggable via the test-patch.d directory, with checkstyle and shellcheck tests included * JIRA comments now use much more markup to improve readability * test-patch.sh now supports either a file name, a URL, or a JIRA issue as input in developer mode * If part of the patch testing code is changed, test-patch.sh will now re-executing itself using those new versions. was: * test-patch.sh now has new output that is different than the previous versions * test-patch.sh is now pluggable via the test-patch.d directory, with checkstyle and shellcheck tests included * JIRA comments now use much more markup to improve readability * test-patch.sh now supports either a file name, a URL, or a JIRA issue as input in developer mode rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch, HADOOP-11746-06.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Status: Patch Available (was: Open) rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Status: Open (was: Patch Available) rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-05.patch -05: * more major code cleanup: renamed quite a few functions, removed more cruft, etc * shelldoc capability docs * added a different table for test failures * slightly changed the overall +/-1 output * fixed some issues with relative paths * reworked some branch detection stuff * reworked the failed test output list to remove the org.apache part Need someone to review this now rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch, HADOOP-11746-05.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-04.patch -04: * effectively same as -03, but adds a subtask section (which in some releases 2x and 3x the changes file!) and removes the decrement version thing rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: (was: HADOOP-11746-04.patch) rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Release Note: * test-patch.sh now has new output that is different than the previous versions * test-patch.sh is now pluggable via the test-patch.d directory, with checkstyle and shellcheck tests included * JIRA comments now use much more markup to improve readability * test-patch.sh now supports either a file name, a URL, or a JIRA issue as input in developer mode Hadoop Flags: Incompatible change rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-04.patch -04: * *MUCH* better branch and issue detection on patches * checkstyle sort of works now! * add missing license files * some general code cleanup * JIRA comments are now formatted correctly * console comments had some display issues fixed * Removed a lot of if jenkins this, if not jenkins this, simplifying the code paths * in addition to files, test-patch.sh now supports issues and http(s) URLs which will automatically download patches off of JIRA * give some hints about execution times while running too * fixed some issues around running tests To-do: * Test on Jenkins * Do more broken patch tests * Any other requests? rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Status: Patch Available (was: Open) rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch, HADOOP-11746-04.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-03.patch -03: * Beginnings of plug-in support, including some very alpha checkstyle and shellcheck plug-ins * --debug support * Branch detection for --dirty-workspace and the ability to work on branches that aren't trunk (note: must conform to HowToContribute wiki!) * Some more output cleanup/enhancements * re-arrange/label the usage/help message output * re-order the basic tests a bit * include some status/timings for things that are normally hidden * imported some routines from hadoop-funcions.sh, so the start of some shelldoc.py support Any hints on how I can start testing under jenkins? How do I duplicate what we currently have? rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch, HADOOP-11746-02.patch, HADOOP-11746-03.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[
https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-11746:
--
Attachment: HADOOP-11746-02.patch
-02:
* still very little new functionality, except that --dirty-workspace is now
working for me.
* console output cleanup
* fix the /tmp default for the patch scratch space to be
/tmp/(projname)-test-patch/pid to allow for simultaneous test patch runs
* add in a timer to show how long various steps take (as requested by
[~raviprak] )
* Fix more we should have a var rather than hard code this binary issues
* just skip the findbugs test if findbugs isn't installed
* send the mvn install run to a log file rather than dumping it to the screen
* fix some of the backslash indentation problems generated by the autoformatter
Current console output now looks like:
{code}
w$ dev-support/test-patch.sh --dirty-workspace /tmp/H1
Running in developer mode
/tmp/Hadoop-test-patch/54448 has been created
===
===
Testing patch for H1.
===
===
-1 overall
| Vote | Subsystem | Comment
| +1 |@author | 00m 00s | The patch does not contain any
| @author tags.
| -1 | tests included | 00m 00s | The patch doesn't appear to include
| any new or modified tests. Please
| justify why no new tests are needed
| for this patch. Also please list what
| manual steps were performed to verify
| this patch.
| +1 | javac | 04m 30s | There were no new javac warning
| messages.
| +1 |javadoc | 06m 15s | There were no new javadoc warning
| messages.
| +1 |eclipse:eclipse | 00m 24s | The patch built with eclipse:eclipse.
| -1 | release audit | 00m 05s | The applied patch generated 1
release
| audit warnings.
===
===
Finished build.
===
===
{code}
Note the always centered text and the column wrap on the output. :D
rewrite test-patch.sh
-
Key: HADOOP-11746
URL: https://issues.apache.org/jira/browse/HADOOP-11746
Project: Hadoop Common
Issue Type: Test
Components: build, test
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Allen Wittenauer
Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch,
HADOOP-11746-02.patch
This code is bad and you should feel bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-01.patch -01: * start preparing for set -euo pipefail mode (hopeful thinking) * complain if JAVA_HOME isn't found since some of the maven tests fail if it isn't defined * JIRA output now uses more markup, because tables are cool * rework the generic console output header bits to auto-center * rework the error report based upon the new JIRA output, including splitting it out into its own outputter * add a few helper routines to auto center text, etc. * only find the changed modules once instead of twice * run it through a shell code formatter, but I really hate how it handles backslashed lines * lots of oh we have a variable for this binary, but let's ignore it and call whatever is in the path anyway fixes * fix a few inconsistencies in how we checked and reported errors * updated quite a few things to modern bash semantics * local and lowercase'd all of the local vars * removed forrest bits I haven't done a full run yet, never mind against Jenkins, but this one might actually work. ;) Still a lot to do, including making it pluggable and moving some of these bits into that plug-in dir. rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch, HADOOP-11746-01.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Issue Type: Test (was: Bug) rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11746) rewrite test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-11746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11746: -- Attachment: HADOOP-11746-00.patch -00: * initial pass that just fixes the vast majority of shellcheck errors rewrite test-patch.sh - Key: HADOOP-11746 URL: https://issues.apache.org/jira/browse/HADOOP-11746 Project: Hadoop Common Issue Type: Test Components: build, test Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Attachments: HADOOP-11746-00.patch This code is bad and you should feel bad. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
