[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083573#comment-16083573 ] Peter Vary commented on HIVE-15051: --- I am happy with the current wording! Thanks [~leftylev]! > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Fix For: 3.0.0 > > Attachments: beeline.out, HIVE-15051.02.patch, HIVE-15051.patch, > Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083050#comment-16083050 ] Lefty Leverenz commented on HIVE-15051: --- Thanks for the explanations, [~pvary]. I used your own wording and tinkered a bit with paragraph 3 -- feel free to re-edit whether for meaning or for style preferences. * [Running Yetus | https://cwiki.apache.org/confluence/display/Hive/Running+Yetus] > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Fix For: 3.0.0 > > Attachments: beeline.out, HIVE-15051.02.patch, HIVE-15051.patch, > Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080078#comment-16080078 ] Peter Vary commented on HIVE-15051: --- Thanks for the review [~leftylev]! Added the fix version, thanks for pointing that out! My answers: ??What does this phrase mean in paragraph 3: "... no full refactoring is required for change ..."? I guess what puzzles me is the "for change" part.?? I was trying to emphasize that Yetus checks only the changed part of the code. So if the unchanged part contains errors, then Yetus will not report it. I am *very* open for better wording :D ??Why did you omit maven and javac from the Hive personality list, although they are in personality_plugins (dev-support/hive-personality.sh)?? They are needed for running Yetus: - Maven tells Yetus, that it should use maven to build the the code. - Javac tells Yetus, how to generate the compile output Thanks, Peter > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Fix For: 3.0.0 > > Attachments: beeline.out, HIVE-15051.02.patch, HIVE-15051.patch, > Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078925#comment-16078925 ] Lefty Leverenz commented on HIVE-15051: --- Your doc looks good, [~pvary]. I made some trivial edits and added the release number. By the way, this jira needs a fix version -- 3.0.0. Two questions: # What does this phrase mean in paragraph 3: "... no full refactoring is required for change ..."? I guess what puzzles me is the "for change" part. # Why did you omit maven and javac from the Hive personality list, although they are in personality_plugins (dev-support/hive-personality.sh)? Thanks. > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.02.patch, HIVE-15051.patch, > Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064962#comment-16064962 ] Peter Vary commented on HIVE-15051: --- [~leftylev]: Could you please review the docs here: https://cwiki.apache.org/confluence/display/Hive/Running+Yetus Thanks, Peter > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.02.patch, HIVE-15051.patch, > Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16052706#comment-16052706 ] Peter Vary commented on HIVE-15051: --- Thanks Lefty, Will check, and find a place sometime after next week. Thanks, Peter > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.02.patch, HIVE-15051.patch, > Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16052672#comment-16052672 ] Lefty Leverenz commented on HIVE-15051: --- Should this be documented in the wiki? If so, it belongs somewhere in the testing docs: * [Hive Testing Documents | https://cwiki.apache.org/confluence/display/Hive/TestingDocs] > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.02.patch, HIVE-15051.patch, > Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16052015#comment-16052015 ] Peter Vary commented on HIVE-15051: --- Thanks Zoltán for the commit. I would love to see it in work! :) Have you seen the rat errors in the patchprocess directory recently? Sadly I did not have time to take a look at this recently :( > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.02.patch, HIVE-15051.patch, > Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16051972#comment-16051972 ] Zoltan Haindrich commented on HIVE-15051: - pushed to master. Thank you [~pvary] for looking into this! ...it would be great to have this in action. > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.02.patch, HIVE-15051.patch, > Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[
https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034905#comment-16034905
]
Zoltan Haindrich commented on HIVE-15051:
-
[~pvary] I've tried to use it to validate the integrity of one of my pending
patches...and it seems like it identified the files inside the {{patchprocess}}
directory as sources which miss the asf header.
Anyway...I don't want to hold up this thing much longer...the most usefull
thing would be to have this activated...javadoc errors are coming back - after
just a few days they've got fixed...
> Test framework integration with findbugs, rat checks etc.
> -
>
> Key: HIVE-15051
> URL: https://issues.apache.org/jira/browse/HIVE-15051
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: beeline.out, HIVE-15051.02.patch, HIVE-15051.patch,
> Interim.patch, ql.out
>
>
> Find a way to integrate code analysis tools like findbugs, rat checks to
> PreCommit tests, thus removing the burden from reviewers to check the code
> style and other checks which could be done by code.
> Might worth to take a look on Yetus, but keep in mind the Hive has a specific
> parallel test framework.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16030940#comment-16030940 ] Peter Vary commented on HIVE-15051: --- Test failures are not related :) [~kgyrtkirk]: Could you please take a look at it? Thanks, Peter > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.02.patch, HIVE-15051.patch, > Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[
https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16030355#comment-16030355
]
Hive QA commented on HIVE-15051:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12870449/HIVE-15051.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10791 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=145)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5480/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5480/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5480/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12870449 - PreCommit-HIVE-Build
> Test framework integration with findbugs, rat checks etc.
> -
>
> Key: HIVE-15051
> URL: https://issues.apache.org/jira/browse/HIVE-15051
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: beeline.out, HIVE-15051.02.patch, HIVE-15051.patch,
> Interim.patch, ql.out
>
>
> Find a way to integrate code analysis tools like findbugs, rat checks to
> PreCommit tests, thus removing the burden from reviewers to check the code
> style and other checks which could be done by code.
> Might worth to take a look on Yetus, but keep in mind the Hive has a specific
> parallel test framework.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[
https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022881#comment-16022881
]
Peter Vary commented on HIVE-15051:
---
[~kgyrtkirk]:
- Preinstalling artifacts / running only changed modules: I will check how the
Hadoop/HBase etc projects solve this problem. Running build only on the changed
modules helps in reducing the runtime. Your suggested solution could be a good
idea too.
- Clean workspace: By design YETUS can run on dirty workspace if needed
{{./dev-support/test-patch.sh --dirty-workspace}}. I do not really know why
would you do it, but at least you can! :) :)
Thanks for the review! Will take a look at the stuff.
Peter
> Test framework integration with findbugs, rat checks etc.
> -
>
> Key: HIVE-15051
> URL: https://issues.apache.org/jira/browse/HIVE-15051
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out
>
>
> Find a way to integrate code analysis tools like findbugs, rat checks to
> PreCommit tests, thus removing the burden from reviewers to check the code
> style and other checks which could be done by code.
> Might worth to take a look on Yetus, but keep in mind the Hive has a specific
> parallel test framework.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[
https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022815#comment-16022815
]
Zoltan Haindrich commented on HIVE-15051:
-
[~pvary]: There is some issue with the current working of this patch...if I
don't have hive already installed at the local {{.m2}} it doesn't really work -
because it refers to some hive artifacts which are not yet installed...I don't
entirely understand the concept behind executing {{mvn install}} only in just
the changed modules ( this seems to me an interesting idea to do in a multi
module project ).
Anyway adding {{mvn install -DskipTests -DskipSparkTests}} to the test-patch.sh
have fixed this for me - leaving this out may result in unrelated compile
errors...which might be misleading.
note: the usage of the tool requires a clean workspace; however the cleanup is
left for the user...which is a bit odd; or this is normal behaviour?
> Test framework integration with findbugs, rat checks etc.
> -
>
> Key: HIVE-15051
> URL: https://issues.apache.org/jira/browse/HIVE-15051
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out
>
>
> Find a way to integrate code analysis tools like findbugs, rat checks to
> PreCommit tests, thus removing the burden from reviewers to check the code
> style and other checks which could be done by code.
> Might worth to take a look on Yetus, but keep in mind the Hive has a specific
> parallel test framework.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022607#comment-16022607 ] Peter Vary commented on HIVE-15051: --- They might use a different version on the pre-commit jenkins job then. Good to know! Thanks, Peter > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[
https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022580#comment-16022580
]
Zoltan Haindrich commented on HIVE-15051:
-
interesting...they might be using that for some reason...the source doesn't
reference it...
I've seen it in the link you posted earlier:
https://issues.apache.org/jira/browse/YARN-6363?focusedCommentId=15937570&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15937570
- somewhere at the end there is a note: {{Powered by Apache Yetus
0.5.0-SNAPSHOT http://yetus.apache.org}}
yes...it could simplify the wrapper a bit :) I'll continue commenting on the
new ticket then! :D
> Test framework integration with findbugs, rat checks etc.
> -
>
> Key: HIVE-15051
> URL: https://issues.apache.org/jira/browse/HIVE-15051
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out
>
>
> Find a way to integrate code analysis tools like findbugs, rat checks to
> PreCommit tests, thus removing the burden from reviewers to check the code
> style and other checks which could be done by code.
> Might worth to take a look on Yetus, but keep in mind the Hive has a specific
> parallel test framework.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022565#comment-16022565 ] Peter Vary commented on HIVE-15051: --- To be honest I was thinking about using the snapshot version (by downloading the latest Yetus in the wrapper), but decided against it to use a development version of YETUS in "production" :) Where did you find references of 0.5.0-SNAPSHOT in hadoop. I did some digging again, and it seems to me that they are using 0.4.0: https://github.com/apache/hadoop/blob/trunk/dev-support/bin/yetus-wrapper#L76 I might miss something... Docker image would be good. It could help with the stuff which is done now manually hidden in yetus-wrapper.sh and could help with installing findbugs as well. Created a followup jira for it: - HIVE-16749 Run YETUS in Docker container > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022527#comment-16022527 ] Zoltan Haindrich commented on HIVE-15051: - [~pvary] it seems to me that the hadoop guys are using some docker containers to execute yetusand somehow they are already using 0.5.0-SNAPSHOT...I suspect that they have faced similar issues deploying/using yetus - like missing patches. I think it would be great to containerize the yetus runner. That way the current extra "patching" could be described in the Dockerfile. hadoop yetus image: https://github.com/apache/hadoop/blob/trunk/dev-support/bin/create-release#L457 https://github.com/apache/hadoop/blob/trunk/dev-support/docker/Dockerfile what do you think about it? > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022489#comment-16022489 ] Peter Vary commented on HIVE-15051: --- Created followup jiras: - HIVE-16747 Remove YETUS*.sh files after a YETUS release - HIVE-16748 Integreate YETUS to Pre-Commit Thanks [~thejas] for the review! > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022156#comment-16022156 ] Thejas M Nair commented on HIVE-15051: -- +1 Can you also please create a follow up jira to remove the Yetus*.sh files when the new release is out for yetus ? > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019993#comment-16019993 ] Peter Vary commented on HIVE-15051: --- Test failures are not related. Any more comments? > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[
https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16012540#comment-16012540
]
Hive QA commented on HIVE-15051:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12868317/HIVE-15051.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10717 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[table_nonprintable]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3]
(batchId=97)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
(batchId=97)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5282/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5282/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5282/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12868317 - PreCommit-HIVE-Build
> Test framework integration with findbugs, rat checks etc.
> -
>
> Key: HIVE-15051
> URL: https://issues.apache.org/jira/browse/HIVE-15051
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: beeline.out, HIVE-15051.patch, Interim.patch, ql.out
>
>
> Find a way to integrate code analysis tools like findbugs, rat checks to
> PreCommit tests, thus removing the burden from reviewers to check the code
> style and other checks which could be done by code.
> Might worth to take a look on Yetus, but keep in mind the Hive has a specific
> parallel test framework.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011903#comment-16011903 ] Peter Vary commented on HIVE-15051: --- When investigating the Vectorization issue, I have identified this: YETUS-506. > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, Interim.patch, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16007002#comment-16007002 ] Thejas M Nair commented on HIVE-15051: -- [~pvary] Where would the details of the findbugs result (or other steps) show up ? Would it be accessible via jenkins job ? What are remaining steps to get this as part of our precommit testing ? Thanks again for looking into this! > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15711603#comment-15711603 ] Peter Vary commented on HIVE-15051: --- Posted the first version of the findbugs plugin patch for review to the Yetus project (see: YETUS-471) > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: beeline.out, ql.out > > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648050#comment-15648050 ] Peter Vary commented on HIVE-15051: --- I think we have to think about which plugin to turn on, so we get valuable help for the resources we spend. Here are the running times that I have measured: The run times are when the change is in beeline in seconds: - maven install: 28 + 26 - checkstyle: ? - findbugs: 30 + 30 - javadoc: 17 + 16 - cleanup: 11 - rat: 20 Total time: ~ 4 mins The same patch when the change is in ql: - maven install: 75 + 90 - checkstyle: ? - findbugs: 107 + 124 - failed because of memory, so needs some tweaking - javadoc: 45 + 48 - failed because of the wrong javadoc comments - cleanup: 17 - rat: 26 Total time: ~10 mins > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[
https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648020#comment-15648020
]
Peter Vary commented on HIVE-15051:
---
Made a minimal patch for beeline module, here is the test result:
{code}
Total Elapsed time: 3m 49s
-1 overall
_ _ __
| ___|_ _(_) |_ _ _ __ ___| |
| |_ / _` | | | | | | '__/ _ \ |
| _| (_| | | | |_| | | | __/_|
|_| \__,_|_|_|\__,_|_| \___(_)
| Vote | Subsystem | Runtime | Comment
| +1 |hivetest4tests | 0m 00s| The patch appears to include 1 new
| ||| or modified test files.
| +1 | @author | 0m 00s| The patch does not contain any
| ||| @author tags.
| +1 |mvninstall | 0m 28s| passed
| +1 |checkstyle | 0m 16s| passed
| 0 | findbugs | 0m 31s| beeline in has 65 extant Findbugs
| ||| warnings.
| +1 | javadoc | 0m 17s| passed
| +1 |mvninstall | 0m 26s| the patch passed
| -1 |checkstyle | 0m 16s| beeline: The patch generated 9 new +
| ||| 0 unchanged - 0 fixed = 9 total (was
| ||| 0)
| -1 |whitespace | 0m 00s| The patch has 1 line(s) that end in
| ||| whitespace. Use git apply
| ||| --whitespace=fix.
| -1 |whitespace | 0m 00s| The patch 1 line(s) with tabs.
| +1 | xml | 0m 01s| The patch has no ill-formed XML
| ||| file.
| -1 | findbugs | 0m 39s| beeline generated 2 new + 65
| ||| unchanged - 0 fixed = 67 total (was
| ||| 65)
| -1 | javac | 0m 39s| beeline generated 2 new + 65
| ||| unchanged - 0 fixed = 67 total (was
| ||| 65)
| -1 | javadoc | 0m 16s| beeline generated 1 new + 52
| ||| unchanged - 0 fixed = 53 total (was
| ||| 52)
| -1 |asflicense | 0m 20s| The patch generated 15 ASF License
| ||| warnings.
| || 3m 49s|
Reason | Tests
FindBugs | module:beeline
| add.CheckError.get() may expose internal representation by
returning CheckError.data At CheckError.java:by returning CheckError.data At
CheckError.java:[line 18]
| Unwritten field:CheckError.java:[line 18]
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle xml
|
| uname | Darwin pvary.local 15.3.0 Darwin Kernel Version 15.3.0: Thu Dec 10
18:40:58 PST 2015; root:xnu-3248.30.4~1/RELEASE_X86_64 x86_64 |
| Build tool | maven |
| Personality |
/Users/petervary/dev/upstream/hive/dev-support/hive-personality.sh |
| git revision | / 0757a11 |
| Default Java | 1.8.0_92 |
| findbugs | v3.0.1 |
| checkstyle | /private/tmp/yetus-24765.18993/diff-checkstyle-beeline.txt |
| whitespace | /private/tmp/yetus-24765.18993/whitespace-eol.txt |
| whitespace | /private/tmp/yetus-24765.18993/whitespace-tabs.txt |
| findbugs | /private/tmp/yetus-24765.18993/new-findbugs-beeline.html |
| javac | /private/tmp/yetus-24765.18993/new-findbugs-beeline.html |
| javadoc | /private/tmp/yetus-24765.18993/diff-javadoc-javadoc-beeline.txt |
| asflicense | /private/tmp/yetus-24765.18993/patch-asflicense-problems.txt |
| modules | C: beeline U: beeline |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
Finished build.
{code}
> Test framework integration with findbugs, rat checks etc.
> -
>
> Key: HIVE-15051
> URL: https://issues.apache.org/jira/browse/HIVE-15051
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Peter Vary
>Assignee: Peter Vary
>
> Find a way to integrate code analysis tools like findbugs, rat checks to
> PreCommit tests, thus removi
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648007#comment-15648007 ] Peter Vary commented on HIVE-15051: --- The current status: I am trying my hands on [Apache Yetus 0.3.0|https://yetus.apache.org/documentation/0.3.0/]. What I like about it: - Several existing tests (asflicense, author, checkstyle, findbugs, javac, javadoc, test4tests, unitveto, whitespace, xml, junit) - It shows changes in errors so we do not have to clean up the code, but new code will be checked. - Used by multiple ASF projects already - so we will be Apache conform using it. - Extensible, so if we decide to add the ptest framework to these test this could be done Created a hive personality and was able to enable most of the plugins. - Asflicense - apache-rat:check - this works flawlessly - Author - Checks for @author tags - this works flawlessly - Checkstyle - There is an existing checkstyle.xml in hive, but this could not be run on a single module only from the root. If I remove the header check from it, then the single module run is possible. After this change the test could be enabled. The current checkstyle configuration needs some serious be improvement. - Findbugs - Needs the findbugs binary installed, and yetus currently restricts checks for standard maven layouts (/src/main/java, /src/test/java) - We use this source layout only in itest modules. This test run could take more time. Need some trivial changes in yetus. - Javac - I have enabled it for now but there might be a better solution for it. It counts the error/warning messages in the mvn install output, so compilation warnings can be highlighted - Javadoc - Could be enabled easily. Currently there are errors in javadocs, where extra params or exceptions left after the method signature is changed. This prevents the calculation for these modules, but I think these errors could be cleaned up by a script. - Test4tests - Should create or own to handle query files as tests - I created one which +1 a patch only if at least one unit or query test is added or changed - Unitveto - Could be enabled easily, but I do not think we want to veto changes in specific files at the moment - Whitespace - Counts the tabs, and the whitespaces at the end of the lines - this works flawlessly - Xml - Checks the formatting of the xml-s - this works flawlessly - JUnit - I think we do not want to use this - we would like to use ptest instead > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15051) Test framework integration with findbugs, rat checks etc.
[ https://issues.apache.org/jira/browse/HIVE-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15616325#comment-15616325 ] Thejas M Nair commented on HIVE-15051: -- [~pvary] Thanks for looking into this. This will be very valuable. Hadoop builds have had this for a while. > Test framework integration with findbugs, rat checks etc. > - > > Key: HIVE-15051 > URL: https://issues.apache.org/jira/browse/HIVE-15051 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Peter Vary >Assignee: Peter Vary > > Find a way to integrate code analysis tools like findbugs, rat checks to > PreCommit tests, thus removing the burden from reviewers to check the code > style and other checks which could be done by code. > Might worth to take a look on Yetus, but keep in mind the Hive has a specific > parallel test framework. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
