On Mon, Jun 22, 2015 at 12:43 PM, Colin P. McCabe <cmcc...@apache.org> wrote:
> You mentioned that "most of our project will be focused on shell > scripts" I guess based on the existing test-patch code. Allen did a > lot of good work in this area recently. I am curious if you evaluated > languages such as Python or Node.js for this use-case. Shell scripts > can get a little... tricky beyond a certain size. On the other hand, > if we are standardizing on shell, which shell and which version? > Perhaps bash 3.5+? > I'll also add that shell is not helpful for a cross-platform set of tooling. I recently added a daemon to Apache Phoenix; an explicit requirement was Windows support. I ended up implementing a solution in python because that environment is platform-agnostic and still systems-y enough. I think this is something this project should seriously consider. -n On Tue, Jun 16, 2015 at 7:55 PM, Sean Busbey <bus...@cloudera.com> wrote: > > I'm going to try responding to several things at once here, so apologies > if > > I miss anyone and sorry for the long email. :) > > > > > > On Tue, Jun 16, 2015 at 3:44 PM, Steve Loughran <ste...@hortonworks.com> > > wrote: > > > >> I think it's good to have a general build/test process projects can > share, > >> so +1 to pulling it out. You should get help from others. > >> > >> regarding incubation, it is a lot of work, especially for something > that's > >> more of an in-house tool than an artifact to release and redistribute. > >> > >> You can't just use apache labs or the build project's repo to work on > this? > >> > >> if you do want to incubate, we may want to nominate the hadoop project > as > >> the monitoring PMC, rather than incubator@. > >> > >> -steve > >> > >> > > Important note: we're proposing a board resolution that would directly > pull > > this code base out into a new TLP; there'd be no incubator, we'd just > > continue building community and start making releases. > > > > The proposed PMC believes the tooling we're talking about has direct > > applicability to projects well outside of the ASF. Lot's of other open > > source projects run on community contributions and have a general need > for > > better QA tools. Given that problem set and the presence of a community > > working to solve it, there's no reason this needs to be treated as an > > in-house build project. We certainly want to be useful to ASF projects > and > > getting them on-board given our current optimization for ASF infra will > > certainly be easier, but we're not limited to that (and our current > > prerequisites, a CI tool and jira or github, are pretty broadly > available). > > > > > > On Tue, Jun 16, 2015 at 10:13 AM, Nick Dimiduk <ndimi...@apache.org> > wrote: > > > >> > >> Since we're tossing out names, how about Apache Bootstrap? It's a > >> meta-project to help other projects get off the ground, after all. > >> > > > > > > There's already a web development framework named Bootstrap[1]. It's also > > used by several ASF projects, so I think it best to avoid the confusion. > > > > The name is, of course, up to the proposed PMC. As a bit of background, > the > > current name Yetus fulfills Allen's desire to have something shell > related > > and my desire to have a project that starts with Y (there are currently > no > > ASF projects that start with Y). The universe of names that fill in these > > two is very small, AFAICT. I did a brief suitability search and didn't > find > > any blockers. > > > > > > On Tue, Jun 16, 2015 at 11:59 AM, Allen Wittenauer <a...@altiscale.com> > > wrote: > > > >> > >> Since a couple of people have brought it up: > >> > >> I think the release question is probably one of the big question > >> marks. Other than tar balls, how does something like this actually get > >> used downstream? > >> > >> For test-patch, in particular, I have a few thoughts on this: > >> > >> Short term: > >> > >> * Projects that want to move RIGHT NOW would modify their > Jenkins > >> jobs to checkout from the Yetus repo (preferably at a well known tag or > >> branch) in one directory and their project repo in another directory. > Then > >> it’s just a matter of passing the correct flags to test-patch. This is > >> pretty much how I’ve been personally running test-patch for about 6 > months > >> now. Under Jenkins, we’ve seen this work with NiFi (incubating) already. > >> > >> * Create a stub version of test-patch that projects could check > >> into their repo, replacing the existing test-patch. This stub version > >> would git clone from either ASF or github and then execute test-patch > >> accordingly on demand. With the correct smarts, it could make sure it > has > >> a cached version to prevent continual clones. > >> > >> Longer term: > >> > >> * I’ve been toying with the idea of (ab)using Java repos and > >> packaging as a transportation layer, either in addition or in > combination > >> with something like a maven plugin. Something like this would clearly > be > >> better for offline usage and/or to lower the network traffic. > >> > > > > It's important that the project follow ASF guidelines on publishing > > releases[2]. So long as we publish releases to the distribution > directory I > > think we'd be fine having folks work off of the corresponding tag. I'm > not > > sure there's much reason to do that, however. A Jenkins job can just as > > easily grab a release tarball as a git tag and we're not talking about a > > large amount of stuff. The kind of build setup that Chris N mentioned is > > also totally doable now that there's a build description DSL for > Jenkins[3]. > > > > For individual developers, I don't see any reason we can't package things > > up as a tool, similar to how findbugs or shellcheck work. We can make OS > > packages (or homebrew for OS X) if we want to make stand alone > installation > > on developer machines real easy. Those same packages could be installed > on > > the ASF build machines, provided some ASF project wanted to make use of > > Yetus. > > > > Having releases will incur some turn around time for when folks want to > see > > fixes, but that's a trade off around release cadence we can work out > longer > > term. > > > > I would like to have one or two projects that can work off of the > bleeding > > edge repo, but we'd have to get that to mesh with foundation policy. My > gut > > tells me we should be able to come up with an agreement that makes such a > > project "part of the development community" but the specifics will have > to > > be worked out. > > > > > > On Tue, Jun 16, 2015 at 4:41 AM, Jonathan Hsieh <j...@cloudera.com> > wrote: > > > >> How would we have to add testing to our pre commit testing? > >> > >> Each project has likely customized their own core commit scripts > (multiple > >> Jvms versions, checkstyle, javadoc exceptions etc.). We should probably > >> soliciit interest from other projects who already have fancy precommit > >> tests beyond HBase/Hadoop too. > >> > > > > I'm not sure if Allen's response above answered this for you. The current > > state of test-patch has a plugin system for adding new tests. It is also > > customizable to handle project idiosyncrasies (atleast the ones we've > seen > > so far, like unspecified module dependencies, multiple projects per repo, > > or use of ant). Releases will naturally include docs on how to leverage > > those to customize what a particular project needs tested pre-commit. > > > > Given the nature of the work Yetus is hoping to enable, I think it's safe > > to assume the project community is going to be doing a fair bit of > outreach > > to help show projects outside of HBase/Hadoop how things can work for > them. > > > > > > > > [1]: http://getbootstrap.com/ > > [2]: http://www.apache.org/dev/release.html > > [3]: https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin > > > > -- > > Sean >
