Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-07-11 Thread Sean Busbey
As mentioned on HADOOP-12111, there is now an incubator-style proposal: http://wiki.apache.org/incubator/YetusProposal On Wed, Jun 24, 2015 at 9:41 AM, Sean Busbey bus...@cloudera.com wrote: Hi Folks! Work in a feature branch is now being tracked by HADOOP-12111. On Thu, Jun 18, 2015 at

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-24 Thread Nigel Daley
+1 for a separate project and going directly to TLP if possible (as Hadoop itself did when split out of Nutch) +1 for having language discussions once it's a TLP :-) Cheers, Nigel On Jun 22, 2015, at 1:55 PM, Andrew Purtell apurt...@apache.org wrote: On Mon, Jun 22, 2015 at 1:03 PM, Nick

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-24 Thread Sean Busbey
Hi Folks! Work in a feature branch is now being tracked by HADOOP-12111. On Thu, Jun 18, 2015 at 10:07 PM, Sean Busbey bus...@cloudera.com wrote: It looks like we have consensus. I'll start drafting up a proposal for the next board meeting (July 15th). Once we work out the name I'll submit

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-22 Thread Nick Dimiduk
On Mon, Jun 22, 2015 at 12:43 PM, Colin P. McCabe cmcc...@apache.org wrote: You mentioned that most of our project will be focused on shell scripts I guess based on the existing test-patch code. Allen did a lot of good work in this area recently. I am curious if you evaluated languages such

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-22 Thread Andrew Purtell
On Mon, Jun 22, 2015 at 1:03 PM, Nick Dimiduk ndimi...@gmail.com wrote: On Mon, Jun 22, 2015 at 12:43 PM, Colin P. McCabe cmcc...@apache.org wrote: You mentioned that most of our project will be focused on shell scripts I guess based on the existing test-patch code. Allen did a lot of

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-22 Thread Colin P. McCabe
+1 for making this a separate project. We've always struggled with a lot of forks of the test-patch code and perhaps this project can help create something that works well for multiple projects. Bypassing the incubator seems kind of weird (I didn't know that was an option) but I will let other

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-18 Thread Sean Busbey
It looks like we have consensus. I'll start drafting up a proposal for the next board meeting (July 15th). Once we work out the name I'll submit a PODLINGNAMESEARCH jira to track that we did due diligence on whatever we pick. In the mean time, Hadoop PMC would y'all be willing to host us in a

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-17 Thread Ray Chiang
For words beginning with Y Yale: Mythical animal elephant/boar. After some quick Googling, it's apparently originally documented by Pliny the Elder, so it's got a beer-related connotation too. Downside, might get confused with the university of the same name. Yare: adj for quick/agile/lively

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-17 Thread Steve Loughran
On 17 Jun 2015, at 03:55, Sean Busbey bus...@cloudera.com wrote: The name is, of course, up to the proposed PMC. As a bit of background, the current name Yetus fulfills Allen's desire to have something shell related and my desire to have a project that starts with Y (there are currently no

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-16 Thread Kengo Seki
+1. From its user's viewpoint, recent improvements on test-patch made my work really efficient. For example, quick feedback due to avoiding unnecessary tests, automated build environment setup due to Docker support, automated patch download from JIRA, automated shellcheck and whitespace checker,

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-16 Thread Nick Dimiduk
I think this is a great idea! Having just gone through the process of getting Phoenix up to speed with precommits, it would be really nice to have a place to go other than fork/hack someone else's work. For the same project, I recently integrated its first daemon service. This meant adding a bunch

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-16 Thread Tsuyoshi Ozawa
+1 on the idea. It would be great if tests about dependency management. multiple branches, and distributed environment can be done in the project. One discussion point is how Hadoop depends on Yetus, including the development cycles. It's a good time to rethink what's can be done for making

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-16 Thread Allen Wittenauer
Since a couple of people have brought it up: I think the release question is probably one of the big question marks. Other than tar balls, how does something like this actually get used downstream? For test-patch, in particular, I have a few thoughts on this: Short term:

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-16 Thread Steve Loughran
I think it's good to have a general build/test process projects can share, so +1 to pulling it out. You should get help from others. regarding incubation, it is a lot of work, especially for something that's more of an in-house tool than an artifact to release and redistribute. You can't just

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-16 Thread Sean Busbey
I'm going to try responding to several things at once here, so apologies if I miss anyone and sorry for the long email. :) On Tue, Jun 16, 2015 at 3:44 PM, Steve Loughran ste...@hortonworks.com wrote: I think it's good to have a general build/test process projects can share, so +1 to pulling

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-16 Thread Chris Douglas
+1 A separate project sounds great. It'd be great to have more standard tooling across the ecosystem. As a practical matter, how should projects consume releases? -C On Mon, Jun 15, 2015 at 4:47 PM, Sean Busbey bus...@cloudera.com wrote: Oof. I had meant to push on this again but life got in

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-15 Thread Josh Elser
+1 (Have been talking to Sean in private on the subject -- seems appropriate to voice some public support) I'd be interested in this for Accumulo and Slider. For Accumulo, we've come a far way without a pre-commit build, primarily due to a CTR process. We have seen the repeated questions of

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-15 Thread Chris Nauroth
+1 ZooKeeper is another project that has expressed interest in improving its pre-commit process lately. I understand Allen has had some success applying this to the ZooKeeper build too, with some small caveats around quirks in the build.xml that I think we can resolve. I'm interested in

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-15 Thread Allen Wittenauer
I'm clearly +1 on this idea. As part of the rewrite in Hadoop of test-patch, it was amazing to see how far and wide this bit of code as spread. So I see consolidating everyone's efforts as a huge win for a large number of projects. (esp considering how many I saw suffering from a

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-15 Thread Sean Busbey
thank you for making a more digestible version Allen. :) If you're interested in soliciting feedback from other projects, I created ASF short links to this thread in common-dev and hbase: * http://s.apache.org/yetus-discuss-hadoop * http://s.apache.org/yetus-discuss-hbase While I agree that

Re: [DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-15 Thread Sean Busbey
Oof. I had meant to push on this again but life got in the way and now the June board meeting is upon us. Sorry everyone. In the event that this ends up contentious, hopefully one of the copied communities can give us a branch to work in. I know everyone is busy, so here's the short version of

Re: upstream jenkins build broken?

2015-06-06 Thread Sean Busbey
Hi Folks! After working on test-patch with other folks for the last few months, I think we've reached the point where we can make the fastest progress towards the goal of a general use pre-commit patch tester by spinning things into a project focused on just that. I think we have a mature enough

[DISCUSS] project for pre-commit patch testing (was Re: upstream jenkins build broken?)

2015-06-06 Thread Sean Busbey
Sorry for the resend. I figured this deserves a [DISCUSS] flag. On Sat, Jun 6, 2015 at 10:39 PM, Sean Busbey bus...@cloudera.com wrote: Hi Folks! After working on test-patch with other folks for the last few months, I think we've reached the point where we can make the fastest progress

Re: upstream jenkins build broken?

2015-03-12 Thread Chris Nauroth
TestDataNodeHotSwapVolumes, TestDataNodeVolumeFailure, TestDataNodeVolumeFailureReporting, and TestDataNodeVolumeFailureToleration all remove executable permissions from directories like the one Colin mentioned to simulate disk failures at data nodes. I reviewed the code for all of those, and

Re: upstream jenkins build broken?

2015-03-11 Thread Colin McCabe
Is there a maven plugin or setting we can use to simply remove directories that have no executable permissions on them? Clearly we have the permission to do this from a technical point of view (since we created the directories as the jenkins user), it's simply that the code refuses to do it.

Re: upstream jenkins build broken?

2015-03-11 Thread Chris Nauroth
The only thing I'm aware of is the failOnError option: http://maven.apache.org/plugins/maven-clean-plugin/examples/ignoring-errors .html I prefer that we don't disable this, because ignoring different kinds of failures could leave our build directories in an indeterminate state. For example,

Re: upstream jenkins build broken?

2015-03-11 Thread Colin McCabe
On Wed, Mar 11, 2015 at 2:34 PM, Chris Nauroth cnaur...@hortonworks.com wrote: The only thing I'm aware of is the failOnError option: http://maven.apache.org/plugins/maven-clean-plugin/examples/ignoring-errors .html I prefer that we don't disable this, because ignoring different kinds of

Re: upstream jenkins build broken?

2015-03-11 Thread Sean Busbey
You could rely on a destructive git clean call instead of maven to do the directory removal. -- Sean On Mar 11, 2015 4:11 PM, Colin McCabe cmcc...@alumni.cmu.edu wrote: Is there a maven plugin or setting we can use to simply remove directories that have no executable permissions on them?

upstream jenkins build broken?

2015-03-10 Thread Colin P. McCabe
Hi all, A very quick (and not thorough) survey shows that I can't find any jenkins jobs that succeeded from the last 24 hours. Most of them seem to be failing with some variant of this message: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean)

Re: upstream jenkins build broken?

2015-03-10 Thread Aaron T. Myers
Hey Colin, I asked Andrew Bayer, who works with Apache Infra, what's going on with these boxes. He took a look and concluded that some perms are being set in those directories by our unit tests which are precluding those files from getting deleted. He's going to clean up the boxes for us, but we