Just a reminder that it would be great if a few people could review this patch offering. If we can get HBase-oriented Maven archetypes publicly available (via the Maven Central Repository), it might prove quite helpful to new adopters (or those sites that are considering adoption) of HBase technology.
https://github.com/dvimont/test_hbasearchetypes_readme https://issues.apache.org/jira/browse/HBASE-14877 On Sat, Jan 30, 2016 at 9:50 AM, Daniel Vimont <[email protected]> wrote: > Here is the new markdown version of the README rendered on a github test > page (much easier to read than the raw text was): > > https://github.com/dvimont/test_hbasearchetypes_readme > > On Fri, Jan 29, 2016 at 6:28 AM, Daniel Vimont <[email protected]> > wrote: > >> Hi Nick, >> >> Perhaps my choice of the word "infrastructure" is rather poor -- >> overstating (even *melodramatizing*!) what is there. >> >> I'm hopeful that when you take a look at it, that the structures and the >> reasons that the structures are there will be apparent. (If not, then my >> README text failed one of its initial tests, and needs to be reworked to >> provide further clarifications.) >> >> But, in short, everything that is there, I found necessary to be there in >> order to (1) facilitate proper automatic testing of *all* components of an >> archetype, and (2) provide automation of archetype creation steps that >> would otherwise have to manually be done by a contributor and manually >> verified by a committer. (The folks at the Maven project provide some >> wonderful archetype plugin tools to accomplish a lot of this, and these are >> being leveraged to the hilt; but a few details they don't seem to provide >> for had to be provided via simple bash scripts and xslt transformation.) >> >> At this point, I'll let you take a look at things before I attempt to >> elaborate further... >> >> Thanks, >> >> Dan >> >> On Fri, Jan 29, 2016 at 2:06 AM, Nick Dimiduk <[email protected]> wrote: >> >>> Hi Daniel, >>> >>> Thanks again for picking this one up and really working it! I'm not clear >>> on what new infrastructure is needed. The archetypes would be added as >>> new >>> modules to our existing maven structure, built when existing code is >>> built, >>> tests run when existing code tests are run, and deployed the same. Is >>> that >>> not so? Let me spend some time with your patches (I know I keep saying >>> that...) and come back for a more coherent conversation. >>> >>> -n >>> >>> On Wednesday, January 27, 2016, Daniel Vimont <[email protected]> >>> wrote: >>> >>> > One point to add to this discussion: the intention is to backport this >>> to >>> > make it available in an upcoming 1.x minor release, so that new >>> archetypes >>> > are publicly available via the Maven central repository as soon as >>> > possible. >>> > >>> > As Sean brings up over in the HBASE-14877 JIRA entry >>> > <https://issues.apache.org/jira/browse/HBASE-14877>, we need to >>> determine >>> > "what level of 'done' we're looking for before we do that". >>> > >>> > While I imagine that this will not be anywhere near as involved as the >>> > backporting of the new hbase-spark stuff to branch-1 (see HBASE-14160 >>> > <https://issues.apache.org/jira/browse/HBASE-14160>), this patch does >>> > introduce *new infrastructure* for developing and maintaining new >>> artifacts >>> > (in this case, Maven archetypes), so it certainly does merit some >>> review >>> > before we open up a new JIRA and start adding this functionality to >>> > branch-1. >>> > >>> > On Wed, Jan 27, 2016 at 4:07 PM, Daniel Vimont <[email protected] >>> > <javascript:;>> wrote: >>> > >>> > > Work has been underway for the last month on task HBASE-14877 >>> > > <https://issues.apache.org/jira/browse/HBASE-14877>, the first >>> subtask >>> > of >>> > > HBASE-14876 (in Nick's original wording from that JIRA entry: "To >>> help >>> > > onboard new users, we should *provide maven archetypes for hbase >>> client >>> > > applications*"). >>> > > >>> > > The three originally proposed *Maven archetypes* in HBASE-14876 were >>> > > split into three subtasks, for: >>> > > >>> > > 1. an hbase-client dependent application, >>> > > 2. an hbase-shaded-client dependent application, and >>> > > 3. a map-reduce job example with hbase-server dependency. >>> > > >>> > > *(Note that a fourth subtask has since been added -- for an >>> hbase-spark >>> > > oriented archetype.)* >>> > > >>> > > *The first subtask (HBASE-14877) provides:* >>> > > >>> > > *[a] a new infrastructure* for creation and maintenance of >>> HBase-related >>> > > Maven archetypes, and >>> > > [*b] the first archetype* from the list above (to be used by >>> end-users to >>> > > autogenerate a fully-configured Maven project with hbase-client >>> > dependency >>> > > and fully-functional sample Java code). >>> > > >>> > > >>> > > *It is item [a] above -- a new infrastructure for >>> creation/maintenance of >>> > > archetypes -- that we would particularly like for >>> contributors/committers >>> > > to take a look at, and to offer up comments, questions, suggestions, >>> > etc.* >>> > > The README text in the new hbase-archetypes subdirectory is intended >>> to >>> > be >>> > > a brief yet comprehensive introduction to how this new infrastructure >>> > > works, and how a contributor would add new Maven archetypes in the >>> > future. *(It >>> > > is intended that it be very simple to contribute new archetypes, >>> without >>> > > the need for any special expertise with regard to Maven archetypes in >>> > > general.)* >>> > > >>> > > For ease of access, I'll just paste in the README text below. *(Note >>> that >>> > > this text will soon be reformatted into markdown format, but this is >>> the >>> > > intended content.)* >>> > > >>> > > Besides reviewing this text, you can earn extra gratitude-points if >>> you: >>> > > >>> > > - >>> > > *set up a sandbox environment, * >>> > > - >>> > > *install the latest patch (HBASE-14877-v4.patch >>> > > < >>> > >>> https://issues.apache.org/jira/secure/attachment/12784323/HBASE-14877-v4.patch >>> > >) >>> > > against the HBase master branch, * >>> > > - >>> > > *follow the instructions at the end of the README (below) to >>> generate a >>> > > new project from the archetype (emulating the experience of an >>> > end-user), >>> > > and * >>> > > - *review the results of that process, as well!* >>> > > >>> > > THANKS VERY MUCH, >>> > > Dan Vimont >>> > > >>> > > >>> > > >>> > > >>> > >>> ------------------------------------------------------------------------------- >>> > > *README.txt* >>> > > >>> > > >>> > >>> ------------------------------------------------------------------------------- >>> > > *The hbase-archetypes subproject of hbase provides an infrastructure >>> for* >>> > > *creation and maintenance of Maven archetypes[1] pertinent to HBase. >>> > Upon* >>> > > *deployment to the archetype catalog[2] of the central Maven >>> > > repository[3],* >>> > > *these archetypes may be used by end-user developers to autogenerate >>> > > completely* >>> > > *configured Maven projects (including fully-functioning sample code) >>> > > through* >>> > > *invocation of the archetype:generate goal of the >>> > > maven-archetype-plugin[4].* >>> > > >>> > > *Notes for contributors and committers to the HBase project* >>> > > *==========================================================* >>> > > *The hbase-archetypes project contains a separate subproject for each >>> > > archetype.* >>> > > *The top level components of such a subproject comprise a complete, >>> > > standalone* >>> > > *exemplar Maven project containing (a) a src directory with sample, >>> > fully-* >>> > > *functioning code in the ./main and ./test subdirectories, (b) a >>> pom.xml >>> > > file* >>> > > *defining all required dependencies, and (c) any additional resources >>> > > required* >>> > > *by the exemplar project. (For example, the components of the >>> > > hbase-client-* >>> > > *project consist of (a) sample code ./src/main/.../HelloHBase.java >>> and* >>> > > *./src/test/.../TestHelloHBase.java, (b) a pom.xml file establishing >>> > > dependency* >>> > > *upon hbase-client and test-scope dependency upon >>> hbase-testing-util, and >>> > > (c) a* >>> > > *log4j.properties resource file.)* >>> > > >>> > > *** Overview of the creation of archetypes during the hbase install >>> > > process: *** >>> > > *During the mvn install process, all standalone exemplar projects in >>> the* >>> > > *hbase-archetypes subdirectory are first packaged/tested/installed, >>> and >>> > > then the* >>> > > *following steps are executed in >>> hbase-archetypes/hbase-archetype-builder >>> > > (via* >>> > > *the pom.xml, bash scripts, and xsl templates in that subdirectory):* >>> > > * (1) For each exemplar project, resources are copied (via >>> > > maven-resources-* >>> > > *plugin) and transformed (via xml-maven-plugin xslt functionality) to >>> > the* >>> > > *exemplar project's ./target/build-archetype subdirectory[5].* >>> > > * (2) The script createArchetypes.sh is executed to invoke the >>> > > maven-archetype-* >>> > > *plugin's create-from-project goal within each exemplar project's* >>> > > *./target/build-archetype subdirectory. For each exemplar project, >>> this >>> > > creates* >>> > > *a corresponding Maven archetype in the ./target/build-* >>> > > *archetype/target/generate-sources/archetype subdirectory. (Note that >>> > this >>> > > step* >>> > > *always issues two platform-encoding warnings per archetype, due to >>> > > hard-wired* >>> > > *behavior of the maven-archetype-plugin[6].)* >>> > > * (3) The pom.xml file of each newly-created archetype is copied >>> (via >>> > > maven-* >>> > > *resources-plugin) and transformed (via xml-maven-plugin xslt >>> > > functionality)[7].* >>> > > * (4) The script installArchetypes.sh is executed to install each >>> > > archetype* >>> > > *into the local Maven repository, ready for deployment to the central >>> > > Maven* >>> > > *repository. (Note that installation of an archetype automatically >>> > > includes* >>> > > *invocation of integration-testing prior to install, which performs a >>> > test* >>> > > *generation of a project from the archetype.)* >>> > > >>> > > *** To add a new archetype to the hbase-archetypes collection: *** >>> > > * (1) Create a new subdirectory in hbase-archetypes, populated with >>> a* >>> > > *completely configured Maven project, which will serve as the >>> exemplar >>> > > project* >>> > > *of the new archetype. (It may be most straightforward to simply >>> copy the >>> > > src* >>> > > *and pom.xml components from one of the existing exemplar projects, >>> > > replace the* >>> > > *src/main and src/test code, and modify the pom.xml file's >>> > <dependencies>,* >>> > > *<artifactId>, <name>, and <description> elements.)* >>> > > * (2) Modify the hbase-archetype-builder/pom.xml file: (a) add the >>> new >>> > > exemplar* >>> > > *project to the <modules> element, and (b) add appropriate >>> <execution> >>> > > elements* >>> > > *and <transformationSet> elements within the <plugin> elements (using >>> > the* >>> > > *existing entries from already-existing exemplar projects as a >>> guide).* >>> > > * (3) Add appropriate entries for the new exemplar project to the* >>> > > *createArchetypes.sh and installArchetypes.sh scripts in the >>> > > hbase-archetype-* >>> > > *builder subdirectory (using the existing entries as a guide).* >>> > > >>> > > *** For additional testing/inspection of an archetype in this >>> collection: >>> > > *** >>> > > *Although integration-testing (which is automatically performed for >>> each* >>> > > *archetype during the install process) already performs test >>> generation >>> > of >>> > > a* >>> > > *project from an archetype, it may often be advisable to do further >>> > manual* >>> > > *testing of a newly built and installed archetype, particularly to >>> > examine >>> > > and* >>> > > *test a project generated from the archetype (mimicking the end-user >>> > > experience* >>> > > *of utilizing the archetype). Upon completion of the install process >>> > > outlined* >>> > > *above, all archetypes will have been installed in the local Maven >>> > > repository* >>> > > *and can be tested locally by executing the following:* >>> > > * mvn archetype:generate -DarchetypeCatalog=local* >>> > > *This displays a numbered list of all locally-installed archetypes >>> for >>> > the >>> > > user* >>> > > *to choose from for generation of a new Maven project.* >>> > > *=================================================================* >>> > > *END of notes for contributors and committers to the HBase project* >>> > > >>> > > *Footnotes:* >>> > > *===========================================* >>> > > *[1] http://maven.apache.org/archetype/index.html >>> > > <http://maven.apache.org/archetype/index.html>* >>> > > *[2] http://repo1.maven.org/maven2/archetype-catalog.xml >>> > > <http://repo1.maven.org/maven2/archetype-catalog.xml>* >>> > > *[3] http://search.maven.org/ <http://search.maven.org/>* >>> > > *[4] >>> > > >>> > >>> http://maven.apache.org/archetype/maven-archetype-plugin/generate-mojo.html >>> > > < >>> > >>> http://maven.apache.org/archetype/maven-archetype-plugin/generate-mojo.html >>> > >* >>> > > *[5] Prior to archetype creation, each exemplar project's pom.xml is >>> > > transformed* >>> > > * as follows to make it into a standalone project: resource >>> filtering* >>> > > * replaces ${project.version} with the literal value of the >>> current* >>> > > * project.version; and xslt tranformation (a) copies <groupId> and >>> > > <version>* >>> > > * subelements of <parent> to make them child elements of the root >>> > > element,* >>> > > * and (b) removes the <parent> and <description> elements.* >>> > > *[6] For an explanation of the platform-encoding warning issued >>> during* >>> > > * maven-archetype-plugin processing, see the first answer to this* >>> > > * stackoverflow posting: >>> http://stackoverflow.com/a/24161287/4112172 >>> > > <http://stackoverflow.com/a/24161287/4112172>* >>> > > *[7] Prior to archetype installation, each archetype's pom.xml is >>> > > transformed* >>> > > * as follows: a <project.build.sourceEncoding> subelement with >>> value* >>> > > * 'UTF-8' is added to the <properties> element. This prevents* >>> > > * platform-encoding warnings from being issued when an end-user >>> > > generates* >>> > > * a project from the archetype.* >>> > > >>> > > >>> > >>> ------------------------------------------------------------------------------- >>> > > >>> > >>> >> >> >
