If no one has additional comments on this document i'll go ahead and put it up for a vote... https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66854770
10.01.2017, 12:50, "James Sirota" <[email protected]>: > Hi Larry, > > Thanks for the comments. I beefed up the technical section. How does this > look? > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66854770 > > 0.[FR++].0Metron Release Types > There are two types of Metron releases: > Feature Release (FR) - this is a release that has a significant step forward > in feature capability and is denoted by an upgrade of the second digit > Maintenance Release (MR) - this is a set of patches and fixes that are issued > following the FR and is denoted by an upgrade of the third digit > Release Naming Convention > Metron build naming convention is as follows: 0.[FR].[MR]. We keep the 0. > notation to signify that the project is still under active development and we > will hold a community vote to go to 1.x at a future time > Initiating a New Metron Release > Immediately upon the release of the previous Metron release create two > branches: FR ++ and MR. Create the FR++ branch by incrementing the second > digit like so 0.[FR++].0. Create the MR branch for the previous Metron > release by incrementing the second digit of the previous release like so > 0.[FR].[MR]. All patches to the previous Metron release will be checked in > under the MR branch and where it makes sense also under the FR branch. All > new features will be checked in under the FR branch. > Creating a Feature Release > Step 1 - Initiate a discuss thread > Prior to the release The Release manager should do the following (preferably > a month before the release): > Make sure that the list of JIRAs slated for the release accurately reflects > to reflects the pull requests that are currently in master > Construct an email to the Metron dev board ([email protected]) > which discusses with the community the desire to do a release. This email > should contain the following: > The list of JIRAs slated for the release with descriptions (use the output of > git log and remove all the JIRAs from the last release’s changelog) > A solicitation of JIRAs that should be included with the next release. Users > should rate them as must/need/good to have as well as volunteering. > A release email template is provided here. > Step 2 - Monitor and Verify JIRAs > Once the community votes for additional JIRAs they want included in the > release verify that the pull requests are in before the release, close these > JIRAs and tag them with the release name. All pull requests and JIRAs that > were not slated for this release will go into the next releases. The release > manager should continue to monitor the JIRA to ensure that the timetable is > on track until the release date. On the release date the release manager > should message the Metron dev board ([email protected]) > announcing the code freeze for the release. > Step 3 - Create the Release Branch and Increment Metron version > Create an branch for the release (from a repo cloned from > https://git-wip-us.apache.org/repos/asf/incubator-metron.git). (assuming the > release is 0.[FR++].0 and working from master): > git checkout -b Metron_0.[FR++].0 > git push --set-upstream origin Metron_0.[FR++].0 > File a JIRA to increment the Metron version to 0.[FR++].0. Either do it > yourself or have a community member increment the build version for you. You > can look at a pull request for a previous build to see how this is done. > METRON-533 - Up the version for release DONE > Also, the release manager should have a couple of things set up: > A SVN clone of the repo at > https://dist.apache.org/repos/dist/dev/incubator/metron, We will refer to > this as the dev repo. It will hold the release candidate artifacts > A SVN clone of the repo at > https://dist.apache.org/repos/dist/release/incubator/metron, We will refer to > this as the release repo. It will hold the release artifacts. > Step 4 - Create the Release Candidate > > Now, for each release candidate, we will tag from that branch. Assuming that > this is RC1: > git checkout Metron_0.[FR++].0 && git pull > git tag apache-metron-0.[FR++].0-rc1-incubating > git push origin —tags > > Now, we must grab the release candidate binary from the github releases page > (https://github.com/apache/incubator-metron/releases). In our case, for RC1, > that would be > https://github.com/apache/incubator-metron/archive/apache-metron-0.[FR++].0-rc1-incubating.tar.gz > We will refer to this as the release candidate tarball. > The artifacts for a release (or a release candidate, for that matter) are as > follows: > Release (candidate) Tarball > MD5 hash of the release tarball (md5 apache-metron-Now, we must grab the > release candidate binary from the github releases page > (https://github.com/apache/incubator-metron/releases). In our case, for RC1, > that would be > https://github.com/apache/incubator-metron/archive/apache-metron-0.[FR++].0-rc1-incubating.tar.gz > We will refer to this as the release candidate > tarball.-rc1-incubating.tar.gz > > apache-metron-0.[FR++].0-rc1-incubating.tar.gz.md5) > SHA1 hash of the release tarball (gpg --print-md SHA1 > apache-metron-0.[FR++].0-rc1-incubating.tar.gz > > apache-metron-0.[FR++].0-rc1-incubating.tar.gz.sha) > GPG signature of release tarball by the release manager > Assuming your public code signing key is 0xDEADBEEF, so signing for me would > be: gpg -u 0xDEADBEEF --armor --output > apache-metron-0.[FR++].0-rc1-incubating.tar.gz.asc --detach-sig > apache-metron-0.[FR++].0-rc1-incubating.tar.gz > If you do not know your code signing key as release manager, you must follow > the instructions at https://www.apache.org/dev/release-signing.html#generate > Note: You only need the -u arg if you have more than one public/private key > pair generated. If you have forgotten it, you can find it from the output of > gpg —fingerprint. It’s the last 4 bytes from the key fingerprint. > The LICENSE file from the release tarball > The KEYS file from the release tarball > The DISCLAIMER file from the release tarball > A CHANGES file denoting the changes > We usually construct this by taking the output of git log | grep METRON | sed > 's/\[//g' | sed 's/\]//g' | grep -v “http” and removing the JIRAs from the > previous releases (it’s in time sorted order so this is easy). > > Create a directory named ${VERSION}-RC${RC_NUM}-incubating (in our case, it’s > 0.[FR++].0-RC1-incubating) in the dev repo. Place the artifacts from above > into this directory, add the directory and commit via the subversion client: > svn add 0.[FR++].0-RC1-incubating > svn commit -m "Adding artifacts for Metron 0.[FR++].0-RC1 (incubating)” > Step 5 - Verify the build > Go through the build verification checklist to verify that everything works. > These instructions can be found here: Verifying Builds > Step 6 - Verify licensing > Make sure the release compiles with the following Apache licensing > guidelines: http://www.apache.org/foundation/license-faq.html > Step 7 - Call for a community release vote > Next initiate a [VOTE] threat on the dev list to announce the build vote. The > vote email template can be found here: Build Vote Template. Allow at least 72 > hours for the community to vote on the release. When you get enough votes > close the vote by replying [RESULT][VOTE] to the email thread with the tally > of all the votes > Step 8 - Call for a incubator release vote > Once the community has successfully voted on a release, we must escalate the > vote to the incubator general. The same VOTE thread original email is sent to > [email protected] > > If issues are found with the release and the vote fails, then the vote thread > is closed with a synopsis of the voting results and a new RC is worked on in > the community > If issues are found with the release and the vote succeeds, then we proceed > to cut the release, but should notify the community of the issues via an > email on the dev list with the accompanying JIRA(s) required to correct the > issue(s). > > If no issues are found, then we can cut a release > Again, wait for at least 72 hours and then close the vote. > Step 9 - Stage the finished release > A directory with the name of the version (i.e. 0.3.0) should be made in the > release svn repository > > Collateral from the release candidate in the dev repo should be moved to the > above directory and renamed to remove the rc (e.g. mv > apache-metron-0.3.0-rc1-incubating.tar.gz.sha > apache-metron-0.3.0-incubating.tar.gz.sha) > > Add the directory and commit via the subversion client: > > svn add 0.3.0-RC1-incubating > svn commit -m "Adding artifacts for Metron 0.3.0 (incubating)” > > Remove the old releases from the release repo (only the current version and > the KEYS file should exist there). > Step 14 - Announce build > Send an email out to user@ and dev@ to announce the release along with the > changelog and a word of thanks/praise. > Creating a Maintenance Release > Creation of the Maintenance Release should follow exactly the same set of > steps as creating the Feature Release as outlined above, but with two > exception. First, the version incremented on the maintenance release should > be the MR++ so that the release is named 0.[FR].[MR++]. Second, if a critical > JIRA comes in that requires an immediate patch we may forego steps 2-5 and > immediately cut the MR release. A critical JIRA is something that is either a > security vulnerability or a functional show stopper . > Ensuring Consistency between Feature and Maintenance releases > Being able to maintain the previous release train, with only critical or > important bug fixes and security fixes (generally not new features) for users > who are averse to frequent large changes is very important for production > use. They get stability, while the feature code proceeds as fast as the > community wishes. It is important to assure that all commits to the > maintenance release also get made in the feature branch (if relevant), to > avoid the appearance of regressions in the maintenance branch. The formal > process for assuring this is as follows: > Every maintenance release JIRA should have a corresponding feature JIRA to > make sure that the patch is applied consistently to both branches. The > maintenance JIRA should be cloned and appropriate fix version for the feature > release should be applied. If the fix is not relevant to the feature or > maintenance branch then the submitter must explicitly state this. In general > reviewers should refuse a patch PR unless both feature and maintenance JIRAs > have been created. > The release manager has a responsibility to review all commits to the > maintenance line since last release, and make sure they were duplicated to > the feature branch (unless not relevant, which must also be determined). > > 05.01.2017, 06:32, "larry mccay" <[email protected]>: >> Hi James - >> >> This looks pretty good! >> >> A couple quick comments: >> >> * for step 10 - the KEYS file appears to be provided for each release as >> part of the release candidate itself. While I do see some projects do this, >> I think it is actually best practice to have a single KEYS file in a well >> known place outside of the rc. This decoupling is supposed to make it more >> difficult for an artifact to be tampered with and another KEYS file >> provided. I think most projects that keep the KEYS separate just put them at >> the top level of the ASF mirror area for the project such as at >> https://dist.apache.org/repos/dist/*release*/incubator/metron/ [1]. >> * Related to the above, it seems that in the KEYS file is duplicated at the >> top level of the ASF mirror area for the project as well as in the release >> directory. The one inside the release directory would probably go away by >> addressing the previous comment but it should be noted that there is a >> chance for those two files to be out of sync otherwise. >> * I notice that the DISCLAIMER, LICENSE and CHANGES files are kept outside >> of the archives along with the KEYS file. As long as they are also inside >> the archive it is probably fine but I don't think there is a need for >> LICENSE and DISCLAIMER to be outside. In Knox we do keep the CHANGES >> outside as well so that it can be easily reviewed to determine interest or >> need for upgrade etc. >> * I do also notice that there is no zip archive - you may want to consider >> adding a zip as well. >> * steps 10 and 13 instruct the release manager to stage the rc and the >> final release but there aren't any instructions as to how to do so. Is that >> documented elsewhere? We have specific ant targets to run for >> stage-candidate and promote-release [2]. >> >> Hope this is helpful. >> >> 1. https://www.apache.org/dev/release-signing.html#keys-policy >> 2. >> >> https://cwiki.apache.org/confluence/display/KNOX/Release+Process#ReleaseProcess-Stage >> >> thanks, >> >> --larry >> >> On Wed, Jan 4, 2017 at 7:25 PM, Matt Foley <[email protected]> wrote: >> >>> Hi James, is there a formatted version of this somewhere we can look at? >>> Thanks, >>> --Matt >>> >>> On 1/4/17, 1:53 PM, "James Sirota" <[email protected]> wrote: >>> >>> Revised as per additional comments. Are there more comments? Or can >>> we put this up for a vote? >>> >>> Release Process [DRAFT] >>> Skip to end of metadata >>> Created by James Sirota, last modified just a moment ago Go to start >>> of metadata >>> Metron Release Types >>> There are two types of Metron releases: >>> Feature Release (FR) - this is a release that has a significant step >>> forward in feature capability and is denoted by an upgrade of the second >>> digit >>> Maintenance Release (MR) - this is a set of patches and fixes that are >>> issued following the FR and is denoted by an upgrade of the third digit >>> Release Naming Convention >>> Metron build naming convention is as follows: 0.[FR].[MR]. We keep >>> the 0. notation to signify that the project is still under active >>> development and we will hold a community vote to go to 1.x at a future >>> time >>> Initiating a New Metron Release >>> Immediately upon the release of the previous Metron release create two >>> branches: FR ++ and MR. Create the FR++ branch by incrementing the second >>> digit like so 0.[FR++].0. Create the MR branch for the previous Metron >>> release by incrementing the second digit of the previous release like so >>> 0.[FR].[MR]. All patches to the previous Metron release will be checked in >>> under the MR branch and where it makes sense also under the FR branch. All >>> new features will be checked in under the FR branch. >>> Creating a Feature Release >>> Step 1 - Initiate a discuss thread >>> A week before a new feature release initiate a discuss thread on the >>> Metron dev board announcing the upcoming release and asking the community >>> which still outstanding pull requests people want to include in the next >>> build. >>> Step 2 - Verify JIRA >>> Go through the JIRA and verify that all pull requests that were merged >>> for the upcoming build have JIRAs that are in a closed state and are >>> appropriately labelled with the next build version. >>> Step 3 - Announce a code freeze >>> A day before the release date comment on the discuss thread and let >>> people know that the release is ready. Go through the JIRAs for pull >>> requests that came in during the last week and make sure they are labelled >>> with the next build version. >>> Step 4 - Increment Metron version >>> File a JIRA to increment the Metron version to 0.[FR++].0. Either do >>> it yourself or have a community member increment the build version for >>> you. You can look at a pull request for a previous build to see how this >>> is done >>> Step 5 - Increment build version >>> File a JIRA to increment the Metron version to 0.[FR++].0-RC(n), where >>> RC(n) is the number of the release candidate. Sometimes mistakes occur >>> (builds may get voted down) so it will take multiple RCs to get a build >>> through the vote. The RC(n) will be removed after the successful vote. >>> Step 6 - Verify the build >>> Go through the build verification checklist to verify that everything >>> works. These instructions can be found here: Verifying Builds >>> Step 7 - Verify licensing >>> Make sure the release compiles with the following Apache licensing >>> guidelines: http://www.apache.org/foundation/license-faq.html >>> Step 8 - Generate the changes file >>> Go through the JIRA to generate the changes file, which contains a >>> list of all JIRAs included in the upcoming release. An example of a >>> changes file can be found here: https://dist.apache.org/repos/ >>> dist/dev/incubator/metron/0.3.0-RC1-incubating/CHANGES >>> Step 9 - Tag the RC release >>> Tag the release for the RC in case we need to roll back at some >>> point. An example of a valid tag can be seen here: >>> https://git-wip-us.apache.org/repos/asf?p=incubator-metron. >>> git;a=shortlog;h=refs/tags/apache-metron-0.3.0-rc1-incubating >>> Step 10 - Stage the release >>> The next thing to do is to sign and stage the release including the >>> DISCLAIMER, KEYS, and LICENSE files. A properly signed and staged release >>> can be found here: >>> https://dist.apache.org/repos/dist/dev/incubator/metron/0.3. >>> 0-RC1-incubating/ >>> * Make sure you have your correct profile and keys uploaded to >>> https://id.apache.org/ to properly sign the release and to get access to >>> dist.apache.org >>> Step 11 - Call for a community release vote >>> Next initiate a [VOTE] threat on the dev list to announce the build >>> vote. The vote email template can be found here: Build Vote Template. >>> Allow at least 72 hours for the community to vote on the release. When you >>> get enough votes close the vote by replying [RESULT][VOTE] to the email >>> thread with the tally of all the votes >>> Step 12 - Call for a incubator release vote >>> Upon successful completion of step 11, repeat, but now send the email >>> to the incubator general boards. The email should be identical. Again, >>> wait for at least 72 hours and then close the vote. >>> Step 13 - Stage the finished release >>> If the vote fails at any stage then incorporate feedback, create >>> another RC, and repeat. If both votes pass then stage the resulting >>> artifacts here: https://dist.apache.org/repos/ >>> dist/release/incubator/metron/ >>> Step 14 - Announce build >>> Send a discuss thread to the Metron dev boards announcing the new >>> Metron build >>> Creating a Maintenance Release >>> Creation of the Maintenance Release should follow exactly the same set >>> of steps as creating the Feature Release as outlined above, but with two >>> exception. First, the version incremented on the maintenance release >>> should be the MR++ so that the release is named 0.[FR].[MR++]. Second, if >>> a critical JIRA comes in that requires an immediate patch we may forego >>> steps 2-5 and immediately cut the MR release. A critical JIRA is something >>> that is either a security vulnerability or a functional show stopper . >>> Ensuring Consistency between Feature and Maintenance releases >>> Being able to maintain the previous release train, with only critical >>> or important bug fixes and security fixes (generally not new features) for >>> users who are averse to frequent large changes is very important for >>> production use. They get stability, while the feature code proceeds as >>> fast as the community wishes. It is important to assure that all commits >>> to the maintenance release also get made in the feature branch (if >>> relevant), to avoid the appearance of regressions in the maintenance >>> branch. The formal process for assuring this is as follows: >>> Every maintenance release JIRA should have a corresponding feature >>> JIRA to make sure that the patch is applied consistently to both branches. >>> The maintenance JIRA should be cloned and appropriate fix version for the >>> feature release should be applied. If the fix is not relevant to the >>> feature or maintenance branch then the submitter must explicitly state >>> this. In general reviewers should refuse a patch PR unless both feature >>> and maintenance JIRAs have been created. >>> The release manager has a responsibility to review all commits to the >>> maintenance line since last release, and make sure they were duplicated to >>> the feature branch (unless not relevant, which must also be determined). >>> >>> 20.12.2016, 11:45, "Matt Foley" <[email protected]>: >>> > 1. Agree. Being able to maintain the previous release train, with >>> only critical or important bug fixes and security fixes (generally not new >>> features) for users who are averse to frequent large changes, is very >>> important for production use. They get stability, while the mainline code >>> proceeds as fast as the community wishes. >>> > a. As Kyle points out, it is important to assure that all commits to >>> the maintenance line also get made in the mainline (if relevant), to avoid >>> the appearance of regressions in the mainline. There should be a formal >>> process for assuring this. Possibilities are: >>> > i. The release manager has a responsibility to review all commits to >>> the maint line since last release, and make sure they were duplicated to >>> the mainline (unless not relevant, which must also be determined). >>> > ii. Reviewers refuse to accept PRs for the maint line unless they >>> are twinned with PRs for corresponding changes in the mainline (unless not >>> relevant, which must be stated by the submitter). This should be reflected >>> in Jira practices as well as PR practices. Note Jira is poor at tracking >>> multiple “Fix Version/s” values (due to the ambiguous use of “Fix version” >>> to mean both “target version” and “done version”). Most teams just clone >>> jira tickets for multiple target releases. >>> > 2. Agree. Being a release manager is a significant commitment of >>> both time and care, and should be rotated around; both for the benefit of >>> the individuals involved and so that at least 2 or 3 people are deeply >>> familiar with the process at any given time. >>> > --Matt >>> > >>> > On 12/20/16, 8:15 AM, "James Sirota" <[email protected]> wrote: >>> > >>> > You are correct. This thread is about the release process: >>> > https://cwiki.apache.org/confluence/pages/viewpage. >>> action?pageId=66854770 >>> > >>> > Does anyone have additional opinions on this? >>> > >>> > 1. Maintenance release would just contain patches to the >>> existing release. Feature release would contain everything, including >>> patches and new features. >>> > 2. The intention is to rotate the build manager. I did it for >>> the first few releases, then Casey did it for the next few releasees, >>> someone else will probably do it for the next few releases, etc... >>> > >>> > Does this seem reasonable to everyone? >>> > >>> > Thanks, >>> > James >>> > >>> > 18.12.2016, 18:15, "Kyle Richardson" <[email protected] >>> >: >>> > > I think this thread got commingled with the discussion on >>> Coding >>> > > Guidelines. The wiki page on the Release Process is at >>> > > https://cwiki.apache.org/confluence/pages/viewpage. >>> action?pageId=66854770. >>> > > >>> > > Overall, a really informative document. Thanks for pulling >>> this together. >>> > > Two questions: >>> > > >>> > > 1) I'm a little confused about how the feature release and >>> maintenance >>> > > release branches are going to work. Is the idea that all PRs >>> will be merged >>> > > into master and then also be committed to a FR++ or a MR++ >>> branch (or maybe >>> > > even both)? >>> > > >>> > > 2) Are these steps to be taken by a release manager only or is >>> the >>> > > intention that other committers or PMC members rotate through >>> this >>> > > responsibly? Just curious. I actually kind of like the idea of >>> shuffling >>> > > the duty every now and then to avoid burnout by one person. >>> > > >>> > > -Kyle >>> > > >>> > > On Fri, Dec 16, 2016 at 1:31 PM, James Sirota < >>> [email protected]> wrote: >>> > > >>> > >> fixed the link and made one addition that a qualified >>> reviewer is a >>> > >> committer or PPMC member >>> > >> >>> > >> 16.12.2016, 11:07, "[email protected]" <[email protected]>: >>> > >> > Right, I agree. That change looks good to me. >>> > >> > >>> > >> > Looks like the Log4j levels links is broken too. >>> > >> > >>> > >> > For a broken travis - how about "If somehow the tests get >>> into a failing >>> > >> > state on master (such as by a backwards incompatible >>> release of a >>> > >> > dependency) only pull requests intended to rectify master >>> may be merged, >>> > >> > and the removal or disabling of any tests must be +1'd by >>> two reviewers." >>> > >> > >>> > >> > Also, reading through this, should there should be a >>> delineation between >>> > >> a >>> > >> > "reviewer" and somebody who has the ability to vote/+1 a >>> PR? Unless I'm >>> > >> > missing something, right now it looks open to anybody. >>> > >> > >>> > >> > Jon >>> > >> > >>> > >> > On Fri, Dec 16, 2016 at 12:48 PM Nick Allen < >>> [email protected]> wrote: >>> > >> > >>> > >> > Personally, I don't think it matters who merges the pull >>> request. As long >>> > >> > as you meet the requirements for code review, then anyone >>> should be able >>> > >> to >>> > >> > merge it. In fact, I'd rather have the person who knows >>> most about the >>> > >> > change actually merge it into master to ensure that it goes >>> smoothly. >>> > >> > >>> > >> > On Fri, Dec 16, 2016 at 12:15 PM, James Sirota < >>> [email protected]> >>> > >> wrote: >>> > >> > >>> > >> >> Jon, for #2 I changed it to: A committer may merge their >>> own pull >>> > >> request, >>> > >> >> but only after a second reviewer has given it a +1. >>> > >> >> >>> > >> >> 16.12.2016, 10:07, "[email protected]" <[email protected]>: >>> > >> >> > I made some minor changes to the doc - check out the >>> history >>> > >> >> > <https://cwiki.apache.org/confluence/pages/ >>> > >> viewpreviousversions.action? >>> > >> >> pageId=61332235> >>> > >> >> > if you have any concerns. >>> > >> >> > >>> > >> >> > Regarding the larger doc - >>> > >> >> > 1. Not everybody can assign JIRAs to themselves. I >>> recall I had to >>> > >> >> request >>> > >> >> > this access, so that should probably be mentioned. >>> > >> >> > 2. "A committer may never merge their own pull request, >>> a second >>> > >> party >>> > >> >> must >>> > >> >> > merge their changes after it has be properly reviewed." >>> > >> >> > - Is this still true/accurate? I heard both ways. >>> > >> >> > 3. "If somehow the tests get into a failing state on >>> master (such as >>> > >> by >>> > >> > >>> > >> > a >>> > >> >> > backwards incompatible release of a dependency) no pull >>> requests may >>> > >> be >>> > >> >> > merged until this is rectified." >>> > >> >> > - Maybe this should get reassessed using the >>> > >> >> > <https://github.com/apache/incubator-metron/pull/383> >>> most >>> > >> >> > <https://github.com/apache/incubator-metron/pull/381> >>> recent >>> > >> >> > <https://issues.apache.org/jira/browse/METRON-601> build >>> > >> >> > <https://issues.apache.org/jira/browse/METRON-597> >>> failures >>> > >> >> > <https://github.com/apache/incubator-metron/pull/380> >>> as a valuable >>> > >> case >>> > >> >> > study. >>> > >> >> > >>> > >> >> > Jon >>> > >> >> > >>> > >> >> > On Fri, Dec 16, 2016 at 11:38 AM James Sirota < >>> [email protected]> >>> > >> >> wrote: >>> > >> >> > >>> > >> >> >> I threw together a draft document for our release >>> process. Would you >>> > >> >> want >>> > >> >> >> to add/change/delete anything? >>> > >> >> >> >>> > >> >> >> ------------------- >>> > >> >> >> Thank you, >>> > >> >> >> >>> > >> >> >> James Sirota >>> > >> >> >> PPMC- Apache Metron (Incubating) >>> > >> >> >> jsirota AT apache DOT org >>> > >> >> > -- >>> > >> >> > >>> > >> >> > Jon >>> > >> >> > >>> > >> >> > Sent from my mobile device >>> > >> >> >>> > >> >> ------------------- >>> > >> >> Thank you, >>> > >> >> >>> > >> >> James Sirota >>> > >> >> PPMC- Apache Metron (Incubating) >>> > >> >> jsirota AT apache DOT org >>> > >> > >>> > >> > -- >>> > >> > Nick Allen <[email protected]> >>> > >> > >>> > >> > -- >>> > >> > >>> > >> > Jon >>> > >> > >>> > >> > Sent from my mobile device >>> > >> >>> > >> ------------------- >>> > >> Thank you, >>> > >> >>> > >> James Sirota >>> > >> PPMC- Apache Metron (Incubating) >>> > >> jsirota AT apache DOT org >>> > >>> > ------------------- >>> > Thank you, >>> > >>> > James Sirota >>> > PPMC- Apache Metron (Incubating) >>> > jsirota AT apache DOT org >>> >>> ------------------- >>> Thank you, >>> >>> James Sirota >>> PPMC- Apache Metron (Incubating) >>> jsirota AT apache DOT org > > ------------------- > Thank you, > > James Sirota > PPMC- Apache Metron (Incubating) > jsirota AT apache DOT org ------------------- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
