[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[
https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438207#comment-16438207
]
Peter Vary commented on HIVE-18883:
---
[~stakiar]: Something should be amiss here. The {{patchprocess}} directory
should be in {{.gitignore}}, so it should not be deleted. It is not telling I
know the answer :), just that it is strange :(
> Add findbugs to yetus pre-commit checks
> ---
>
> Key: HIVE-18883
> URL: https://issues.apache.org/jira/browse/HIVE-18883
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-18883.1.patch, HIVE-18883.2.patch
>
>
> We should enable FindBugs for our YETUS pre-commit checks, this will help
> overall code quality and should decrease the overall number of bugs in Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[
https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437784#comment-16437784
]
Sahil Takiar commented on HIVE-18883:
-
Ok think I found the issue. This patch isn't working on ptest because the
{{patchprocess}} directory gets deleted as soon as Yetus starts to run because
it requires a clean workspace (we could specify the {{--dirty-workspace}}
option, but that seems dangerous). Yetus checks for the Findbugs directories
after it cleans the workspace, so it decides not to run Findbugs.
Looking at other Apache components, they install Findbugs in a docker image and
then set {{FINDBUGS_HOME}}. Unfortunately, we don't follow that model, so I
need to brainstorm an alternative place to install the Findbugs distribution.
It doesn't look like we can install it anywhere in the cloned git repo.
> Add findbugs to yetus pre-commit checks
> ---
>
> Key: HIVE-18883
> URL: https://issues.apache.org/jira/browse/HIVE-18883
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-18883.1.patch, HIVE-18883.2.patch
>
>
> We should enable FindBugs for our YETUS pre-commit checks, this will help
> overall code quality and should decrease the overall number of bugs in Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[
https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435618#comment-16435618
]
Sahil Takiar commented on HIVE-18883:
-
Looks like this needs to be deployed manually, on {{hiveptest-server-upstream}}
the Hive repo under {{/home/hiveptest/hive}} doesn't have this change. Seems
this repo has to be manually updated.
[~pvary], [~szita] how do we usually deploy these types of changes? I noticed
the most recent change is HIVE-18706. Is simply doing a {{git pull}} sufficient?
> Add findbugs to yetus pre-commit checks
> ---
>
> Key: HIVE-18883
> URL: https://issues.apache.org/jira/browse/HIVE-18883
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-18883.1.patch, HIVE-18883.2.patch
>
>
> We should enable FindBugs for our YETUS pre-commit checks, this will help
> overall code quality and should decrease the overall number of bugs in Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[
https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423798#comment-16423798
]
Peter Vary commented on HIVE-18883:
---
[~stakiar]: The patch contains an extra file \{{HIVE-18883.2.patch}}. Otherwise
+1. I can remove the extra file if you want. I do not know how strict are we
about this.
Thanks for taking care of this!!
> Add findbugs to yetus pre-commit checks
> ---
>
> Key: HIVE-18883
> URL: https://issues.apache.org/jira/browse/HIVE-18883
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18883.1.patch, HIVE-18883.2.patch
>
>
> We should enable FindBugs for our YETUS pre-commit checks, this will help
> overall code quality and should decrease the overall number of bugs in Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[ https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423243#comment-16423243 ] Sahil Takiar commented on HIVE-18883: - [~pvary] any comments on this? > Add findbugs to yetus pre-commit checks > --- > > Key: HIVE-18883 > URL: https://issues.apache.org/jira/browse/HIVE-18883 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18883.1.patch, HIVE-18883.2.patch > > > We should enable FindBugs for our YETUS pre-commit checks, this will help > overall code quality and should decrease the overall number of bugs in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[ https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408670#comment-16408670 ] Sahil Takiar commented on HIVE-18883: - [~szita] any ideas on how I can test this beyond running it locally? Or is running it locally sufficient. > Add findbugs to yetus pre-commit checks > --- > > Key: HIVE-18883 > URL: https://issues.apache.org/jira/browse/HIVE-18883 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18883.1.patch, HIVE-18883.2.patch > > > We should enable FindBugs for our YETUS pre-commit checks, this will help > overall code quality and should decrease the overall number of bugs in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[
https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393427#comment-16393427
]
Sahil Takiar commented on HIVE-18883:
-
[~pvary] is there a good way to test this on Hive QA? I think the patch won't
get applied until after it checks for {{FINDBUGS_HOME}}
{code}
Determining needed tests
(Depending upon input size and number of plug-ins, this may take a while)
...
Applying patch to master
Applying the patch:
Fri Mar 9 06:08:08 UTC 2018
cd /data/hiveptest/working/yetus_PreCommit-HIVE-Build-9564
git apply --binary -v --stat --apply -p1
/data/hiveptest/logs/PreCommit-HIVE-Build-9564/yetus/patch
{code}
I've tested this locally and it works.
> Add findbugs to yetus pre-commit checks
> ---
>
> Key: HIVE-18883
> URL: https://issues.apache.org/jira/browse/HIVE-18883
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18883.1.patch, HIVE-18883.2.patch
>
>
> We should enable FindBugs for our YETUS pre-commit checks, this will help
> overall code quality and should decrease the overall number of bugs in Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[
https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392509#comment-16392509
]
Hive QA commented on HIVE-18883:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913661/HIVE-18883.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 12954 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[
https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392478#comment-16392478
]
Hive QA commented on HIVE-18883:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
10s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
45s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 50 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense findbugs xml javac javadoc compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9564/dev-support/hive-personality.sh
|
| git revision | master / 73ccc44 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9564/yetus/patch-asflicense-problems.txt
|
| modules | C: . standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9564/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add findbugs to yetus pre-commit checks
> ---
>
> Key: HIVE-18883
> URL: https://issues.apache.org/jira/browse/HIVE-18883
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18883.1.patch, HIVE-18883.2.patch
>
>
> We should enable FindBugs for our YETUS pre-commit checks, this will help
> overall code quality and should decrease the overall number of bugs in Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[ https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391980#comment-16391980 ] Sahil Takiar commented on HIVE-18883: - Hive QA hasn't run yet, but attaching updated patch: * Addressed Peter's comment * Got FindBugs working for the standalone-metastore > Add findbugs to yetus pre-commit checks > --- > > Key: HIVE-18883 > URL: https://issues.apache.org/jira/browse/HIVE-18883 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-18883.1.patch, HIVE-18883.2.patch > > > We should enable FindBugs for our YETUS pre-commit checks, this will help > overall code quality and should decrease the overall number of bugs in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[
https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391009#comment-16391009
]
Peter Vary commented on HIVE-18883:
---
[~stakiar]: Thanks for the patch.
One minor comment:
* I would prefer to create {{BASE_YETUS_URL}} and {{BASE_FINDBUGS_URL}}
instead of reusing {{BASEURL}}
I think you were not able to find the {{patchprocess}} on pserver because there
we copy the source directory so the patch validation is able to run parallel
with the tests. [~szita] can tell you more about it (or correct me if I am
wrong :))
{code:java|title=testutils/ptest2/src/main/resources/source-prep.vm}
[..]
rm -rf ../yetus_${buildTag}
mkdir ../yetus_${buildTag}
#Wait until git gc finishes in the background before proceeding:
while ! git gc 2>/dev/null; do sleep 1s ; done
cp -R . ../yetus_${buildTag}
[..]{code}
About dockerization we had another jira, but [~kgyrtkirk] stopped to work on,
an nobody else has taken on yet. See: HIVE-16749
> Add findbugs to yetus pre-commit checks
> ---
>
> Key: HIVE-18883
> URL: https://issues.apache.org/jira/browse/HIVE-18883
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18883.1.patch
>
>
> We should enable FindBugs for our YETUS pre-commit checks, this will help
> overall code quality and should decrease the overall number of bugs in Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[
https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390680#comment-16390680
]
Sahil Takiar commented on HIVE-18883:
-
Thanks for the tip [~pvary]! Helped a lot. What do you think of my patch? It
works when running YETUS locally as described in the wiki you linked above.
Implementation:
* This patch just curls the distribution and sets {{FINDBUGS_HOME}} inside the
{{yetus-wrapper.sh}} file
* The code to download the FindBugs distribution is based on the commands to
download the YETUS distribution (downloads FindBugs to the same place as YETUS
is downloaded), with a few modifications
* One thing I couldn't understand is that when I run {{yetus-wrapper.sh}}
locally it downloads everything to the {{patchprocess}} directory, but I
couldn't find the corresponding directory on the ptest server. Any idea where
it is? Does the YETUS distribution get downloaded for each YETUS run? In which
case FindBugs will be downloaded each time too.
Some other notes while working on this:
* It seems most of the other Apache projects that use YETUS use some sort of
YETUS-Docker integration that downloads the FindBugs dependency
* Ideally, we would do the same thing and we could just do {{apt-get install
findbugs}} and set the {{FINDBUGS_HOME}} appropriately (looks like HBase
manually curls the FindBugs installation)
* It looks like each pre-commit run of these Apache projects builds the
Dockerfile, so it downloads the findbugs distro during each run
* So, overall, I think we are still inline with what other projects are doing,
we just don't use docker (e.g. we download the distribution during each
pre-commit run)
> Add findbugs to yetus pre-commit checks
> ---
>
> Key: HIVE-18883
> URL: https://issues.apache.org/jira/browse/HIVE-18883
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18883.1.patch
>
>
> We should enable FindBugs for our YETUS pre-commit checks, this will help
> overall code quality and should decrease the overall number of bugs in Hive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[ https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389113#comment-16389113 ] Peter Vary commented on HIVE-18883: --- Findbugs binary has to be installed on the ptest server and $FINDBUGS_HOME has to be set before starting yetus. Check out this page: [https://cwiki.apache.org/confluence/display/Hive/Running+Yetus] Thanks for taking on this. It is on my TODO list since introducing yetus, but I never seem to find time to do this. If you need any help, just ask. Thanks, Peter > Add findbugs to yetus pre-commit checks > --- > > Key: HIVE-18883 > URL: https://issues.apache.org/jira/browse/HIVE-18883 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > We should enable FindBugs for our YETUS pre-commit checks, this will help > overall code quality and should decrease the overall number of bugs in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks
[ https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388657#comment-16388657 ] Sahil Takiar commented on HIVE-18883: - [~szita], [~pvary] any tips on how to do this? > Add findbugs to yetus pre-commit checks > --- > > Key: HIVE-18883 > URL: https://issues.apache.org/jira/browse/HIVE-18883 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > We should enable FindBugs for our YETUS pre-commit checks, this will help > overall code quality and should decrease the overall number of bugs in Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
