Re: [PROPOSAL] Metron Community

[Matt Foley]

+1



On 1/17/17, 9:27 PM, "James Sirota"  wrote:

Right now we have 2 entries for Metron Community.  One on our Wiki, which I 
think is outdated and should be removed as it no longer offers value.  The 
second is on our website, which is up to date.  So I am proposing we remove the 
wiki entry.



--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Re: [VOTE] Release Process

[Matt Foley]

+1 (non-binding)

BTW, here is a collection of small editorial changes.  Since these are 
editorial rather than substantive, most project teams accept that they can be 
made by a responsible PMC member (such as our esteemed chair :-) without 
re-voting or disrupting a vote in progress.  I suggest we let James make these 
changes without changing the vote, altho of course if anyone who already voted 
+1 feels that correcting these issues would invalidate your vote, please say so.

Step 4: 2nd bullet: Remove or change obsolete references to the github release 
tarball.

Step 6:  “compiles” --> “complies”

Step 7:  “threat” --> “thread”

Introduction, section “Initiating a New Metron Release”
This sentence is almost certainly a cut-and-paste error:
“Create the MR branch for the previous Metron release by incrementing 
the second digit of the previous release like so 0.[FR].[MR].”
I’m not entirely sure what it should read, but the most probable correction 
based on the sentence before it would, I think, be (remove the asterisks):
“Create the MR branch for the previous Metron release by incrementing 
the *third* digit of the previous release like so 0.[FR].[*MR++*].”

At the end, section “Creating a Maintenance Release”
We got clarification on the urgent voting issue from Mentors, but steps 2-5 
aren’t the steps that get waived.  The two sentences:
“Second, if a critical JIRA comes in that requires an immediate patch 
we may forego steps 2-5 and immediately cut the MR release.  By this we mean 
that 3 binding +1 votes are still required, but the 72 hour waiting period can 
be waved.”
Should be changed to:
“Second, if a critical JIRA comes in that requires an immediate patch, 
the votes with three binding +1's are still required, but Step 1 (discussion) 
and Step 2 (Jira collecting and tracking), and the 72 hour waiting periods in 
Steps 7 and 8 can be waived.”

Cheers,
--Matt

On 1/17/17, 8:17 PM, "James Sirota"  wrote:

I made the revisions based on the discuss thread

The document is available here:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66854770

And is also attached for reference to this email. 

Please vote +1, -1, or 0 for neutral.  The vote will last 72 hours. 

Thanks,
James 

-
Metron Release Types
There are two types of Metron releases:
Feature Release (FR) - this is a release that has a significant step 
forward in feature capability and is denoted by an upgrade of the second digit
Maintenance Release (MR) - this is a set of patches and fixes that are 
issued following the FR and is denoted by an upgrade of the third digit
Release Naming Convention
Metron build naming convention is as follows: 0.[FR].[MR].  We keep the 0. 
notation to signify that the project is still under active development and we 
will hold a community vote to go to 1.x at a future time
Initiating a New Metron Release
Immediately upon the release of the previous Metron release create two 
branches: FR ++ and MR.  Create the FR++ branch by incrementing the second 
digit like so 0.[FR++].0.  Create the MR branch for the previous Metron release 
by incrementing the second digit of the previous release like so 0.[FR].[MR].  
All patches to the previous Metron release will be checked in under the MR 
branch and where it makes sense also under the FR branch.  All new features 
will be checked in under the FR branch.
Creating a Feature Release
Step 1 - Initiate a discuss thread
Prior to the release The Release manager should do the following 
(preferably a month before the release):
Make sure that the list of JIRAs slated for the release accurately reflects 
to reflects the pull requests that are currently in master
Construct an email to the Metron dev board 
(dev@metron.incubator.apache.org) which discusses with the community the desire 
to do a release. This email should contain the following:
The list of JIRAs slated for the release with descriptions (use the output 
of git log and remove all the JIRAs from the last release’s changelog)
A solicitation of JIRAs that should be included with the next release. 
Users should rate them as must/need/good to have as well as volunteering.
A release email template is provided here.
Step 2 - Monitor and Verify JIRAs
Once the community votes for additional JIRAs they want included in the 
release verify that the pull requests are in before the release, close these 
JIRAs and tag them with the release name. All pull requests and JIRAs that were 
not slated for this release will go into the next releases.  The release 
manager should continue to monitor the JIRA to ensure that the timetable is on 
track until the release date.  On the release date the release manager should 
message the Metron dev board (dev@metron.incubator.apache.org) announcing the 
code freeze for the release. 
  

[GitHub] incubator-metron issue #397: METRON-627: Add HyperLogLogPlus implementation ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/397
  
Did you verify that these functions run on:
* the profiler
* the parser topology via field transformations
* the enrichment topology via stellar enrichments
* the Stellar REPL

If so, then I'm +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #397: METRON-627: Add HyperLogLogPlus implemen...

[mmiklavc]

GitHub user mmiklavc reopened a pull request:

https://github.com/apache/incubator-metron/pull/397

METRON-627: Add HyperLogLogPlus implementation to Stellar

This PR addresses https://issues.apache.org/jira/browse/METRON-627

Leverages the HLLP implementation from 
https://github.com/addthis/stream-lib/blob/master/src/main/java/com/clearspring/analytics/stream/cardinality/HyperLogLogPlus.java

4 new Stellar functions have been added that allow a user to initialize a 
cardinality estimator, add items, merge estimators, and calculate cardinality 
estimates.

### `HLLP_CARDINALITY`
  * Description: Returns HyperLogLogPlus-estimated cardinality for this set
  * Input:
* hyperLogLogPlus - the hllp set
  * Returns: Long value representing the cardinality for this set

### `HLLP_INIT`
  * Description: Initializes the set
  * Input:
* p (required) - the precision value for the normal set
* sp - the precision value for the sparse set. If sp is not specified 
the sparse set will be disabled.
  * Returns: A new HyperLogLogPlus set

### `HLLP_MERGE`
  * Description: Merge hllp sets together
  * Input:
* hllp1 - first hllp set
* hllp2 - second hllp set
* hllpn - additional sets to merge
  * Returns: A new merged HyperLogLogPlus estimator set

### `HLLP_OFFER`
  * Description: Add value to the set
  * Input:
* hyperLogLogPlus - the hllp set
* o - Object to add to the set
  * Returns: The HyperLogLogPlus set with a new object added

**Note:** Added new library to metron-common pom and added 3 new items to 
dependencies_with_url.csv.

**Testing**

Spun up the Stellar REPL in quick-dev. And verified that the function 
composition is working as expected and returning correct cardinality estimates 
for simple sparse set cases. For example:
```
[Stellar]>>> HLLP_CARDINALITY(HLLP_MERGE( 
HLLP_OFFER(HLLP_OFFER(HLLP_INIT(5, 6), "runnings"), "cool"), 
HLLP_OFFER(HLLP_OFFER(HLLP_INIT(5, 6), "bobsled"), "team")))
4
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mmiklavc/incubator-metron hyperloglog

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/397.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #397


commit afce30539f6996a607e85d3fd35aac5fcb5c19aa
Author: Michael Miklavcic 
Date:   2016-12-15T20:55:39Z

METRON-627: Add HyperLogLogPlus implementation to Stellar

commit 414a3a98976b98a253ab9921720f02c8a7431da2
Author: Michael Miklavcic 
Date:   2017-01-09T17:00:08Z

work in progress commit

commit c7f57a4acbb0ef357c1af9eaa263afea7bc83d9a
Author: Michael Miklavcic 
Date:   2017-01-11T16:58:58Z

Merge with master

commit 90d9659f415404c6c4682289c7bde669c352f517
Author: Michael Miklavcic 
Date:   2017-01-12T20:33:10Z

Refactor, fix statistics output

commit 261e69651d4ae0b99e88e0e4a2c4e7568aa23fcb
Author: Michael Miklavcic 
Date:   2017-01-12T23:17:13Z

METRON-627: Updated with sensible default precision values

commit 9078094dd720d89f64ecf45506ab0c5077aa58a7
Author: Michael Miklavcic 
Date:   2017-01-13T19:17:37Z

METRON-627: Add default init for HLLP_ADD(null, 'val')

commit 9e1ff937fe51841ac2fa3235bf87964cba8a1ae8
Author: Michael Miklavcic 
Date:   2017-01-17T20:09:26Z

Merge branch 'master' into hyperloglog

commit d392f044e330fe273cb0f0b4ff820b4ef1a3595d
Author: Michael Miklavcic 
Date:   2017-01-17T20:33:11Z

METRON-627: Fix Stellar lexer to handle newline at end




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #397: METRON-627: Add HyperLogLogPlus implementation ...

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/397
  
Kick Travis


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #397: METRON-627: Add HyperLogLogPlus implemen...

[mmiklavc]

Github user mmiklavc closed the pull request at:

https://github.com/apache/incubator-metron/pull/397


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] Release Process

[Kyle Richardson]

+1 (binding)

-Kyle

On Tue, Jan 17, 2017 at 11:17 PM, James Sirota  wrote:

> I made the revisions based on the discuss thread
>
> The document is available here:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66854770
>
> And is also attached for reference to this email.
>
> Please vote +1, -1, or 0 for neutral.  The vote will last 72 hours.
>
> Thanks,
> James
>
> -
> Metron Release Types
> There are two types of Metron releases:
> Feature Release (FR) - this is a release that has a significant step
> forward in feature capability and is denoted by an upgrade of the second
> digit
> Maintenance Release (MR) - this is a set of patches and fixes that are
> issued following the FR and is denoted by an upgrade of the third digit
> Release Naming Convention
> Metron build naming convention is as follows: 0.[FR].[MR].  We keep the 0.
> notation to signify that the project is still under active development and
> we will hold a community vote to go to 1.x at a future time
> Initiating a New Metron Release
> Immediately upon the release of the previous Metron release create two
> branches: FR ++ and MR.  Create the FR++ branch by incrementing the second
> digit like so 0.[FR++].0.  Create the MR branch for the previous Metron
> release by incrementing the second digit of the previous release like so
> 0.[FR].[MR].  All patches to the previous Metron release will be checked in
> under the MR branch and where it makes sense also under the FR branch.  All
> new features will be checked in under the FR branch.
> Creating a Feature Release
> Step 1 - Initiate a discuss thread
> Prior to the release The Release manager should do the following
> (preferably a month before the release):
> Make sure that the list of JIRAs slated for the release accurately
> reflects to reflects the pull requests that are currently in master
> Construct an email to the Metron dev board (dev@metron.incubator.apache.
> org) which discusses with the community the desire to do a release. This
> email should contain the following:
> The list of JIRAs slated for the release with descriptions (use the output
> of git log and remove all the JIRAs from the last release’s changelog)
> A solicitation of JIRAs that should be included with the next release.
> Users should rate them as must/need/good to have as well as volunteering.
> A release email template is provided here.
> Step 2 - Monitor and Verify JIRAs
> Once the community votes for additional JIRAs they want included in the
> release verify that the pull requests are in before the release, close
> these JIRAs and tag them with the release name. All pull requests and JIRAs
> that were not slated for this release will go into the next releases.  The
> release manager should continue to monitor the JIRA to ensure that the
> timetable is on track until the release date.  On the release date the
> release manager should message the Metron dev board (
> dev@metron.incubator.apache.org) announcing the code freeze for the
> release.
> Step 3 - Create the Release Branch and Increment Metron version
> Create an branch for the release (from a repo cloned from
> https://git-wip-us.apache.org/repos/asf/incubator-metron.git). (assuming
> the release is 0.[FR++].0 and working from master):
> git checkout -b Metron_0.[FR++].0
> git push --set-upstream origin Metron_0.[FR++].0
> File a JIRA to increment the Metron version to 0.[FR++].0.  Either do it
> yourself or have a community member increment the build version for you.
> You can look at a pull request for a previous build to see how this is
> done.   METRON-533 - Up the version for release DONE
> Also, the release manager should have a couple of things set up:
> A SVN clone of the repo at https://dist.apache.org/repos/
> dist/dev/incubator/metron, We will refer to this as the dev repo.  It
> will hold the release candidate artifacts
> A SVN clone of the repo at https://dist.apache.org/repos/
> dist/release/incubator/metron, We will refer to this as the release
> repo.  It will hold the release artifacts.
> Step 4 - Create the Release Candidate
>
> Now, for each release candidate, we will tag from that branch. Assuming
> that this is RC1:
> git checkout Metron_0.[FR++].0 && git pull
> git tag apache-metron-0.[FR++].0-rc1-incubating
> git push origin —tags
> Now we must create the release candidate tarball. From the apache repo,
> you should run:
>
>  git archive --prefix=apache-metron-0.[FR++].0-rc1-incubating/
>  apache-metron-0.[FR++].0-rc1-incubating | gzip >
>  apache-metron-0.[FR++].0-rc-incubating.tar.gz
>
> We will refer to this as the release candidate tarball. *Note: Per Apache
> policy, the hardware used to create the candidate tarball must be owned by
> the release manager.
> The artifacts for a release (or a release candidate, for that matter) are
> as follows:
> Release (candidate) Tarball
>  MD5 hash of the release tarball (md5 apache-metron-Now, we must grab the
> release candidate 

Re: [PROPOSAL] Reduce Reliance on Ansible for Deployment

[Otto Fowler]

How would this effect deploying new parsers or parsers not shipped?
When I prototyped this I added a monit entry.


On January 17, 2017 at 10:34:32, David Lyle (dlyle65...@gmail.com) wrote:

In our "Dev Guide and Committer Review Guide additions" discussion, we had
a bit of a side discussion about reducing reliance (perhaps to zero) on
Ansible for our installation.

It seemed there was consensus around that idea (if not, please let me
know), so I propose the following steps to get there:

1) Refactor existing Ansible deployment to use the Ambari MPack to install
metron-common, metron-enrichments and metron-parsers.
2) Regenerate quick-dev to leverage the change.
3) Create rpm packages for all deployed components that don't currently
have them.
- Sensor probes
- Sensor stubs
4) Create MPack service defs for the RPMs in (2).
5) Refactor existing Ansible deployment to use the Ambari MPack to install
all services.
6) Regenerate quick-dev to leverage the change.
7) Plan iteration 2 to see if there are other opportunities to reduce our
use of Ansible.

One note: if we decide to go this direction, it'd be helpful if, during the
transition, we stopped adding additional Ansible deployment code.

Thoughts?

Thanks,

-David...

[RESULT] [VOTE] Reporting Issues Wiki

[James Sirota]

The vote passes with 3 binding +1 votes (James, Casey, Nick), 4 non-binding 
+1s, (matt, JJ, Anand, Jon), and a 0 from Kyle 

Thanks everyone for voting 

18.01.2017, 06:15, "Nick Allen" :
> +1 binding
>
> On Thu, Jan 5, 2017 at 5:41 PM, James Sirota  wrote:
>
>>  Based on feedback from the discuss thread.
>>
>>  Please vote +1, -1, or 0. The vote will be open for 72 hours
>>
>>  https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=67635199
>>
>>  All “I have found a bug” issues are considered developer-level issues.
>>  Please report all developer-level issues to dev@metron.incubator.apache.
>>  org. Examples of developer issues would be:
>>  Project fails to compile or is failing unit or integration tests
>>  Project or individual components fail to install
>>  There are error messages or failures in my logs
>>  I need help with coding or extending a specific component
>>  etc...
>>  After discussion of the issue on the JIRA if it is clear that you found a
>>  bug then you should file a JIRA (unless you found a security
>>  vulnerability). Follow up on the mailing lists if you want advice with
>>  respect to workaround or a local fix. Our JIRA is located here.
>>
>>  All “I have a problem” or "How do you use x" issues are usability
>>  issues. If you are an end-user of the product and have a comment or
>>  question then use u...@metron.incubator.apache.org. If you have a
>>  problem and a strong suspicion that you might have found a bug, please
>>  cross-reference dev@metron.incubator.apache.org as well
>>  I don't understand the UI, what does button x do?
>>  What should the output of function x be?
>>  It would be nice if I had feature x along with feature y
>>  etc...
>>
>>  If you found a security-related issue, please report immediately to
>>  secur...@metron.incubator.apache.org. Please adhere to the following
>>  Apache policy found here. DO NOT FILE A JIRA, DO NOT POST ON ANY OTHER BOARD
>>  I can get access to data I should not have access to
>>  I have privileges to do things I should not be allowed to do
>>  I found that this project is susceptible to an exploit
>>  etc...
>>
>>  Please report issues related to the JIRA/Wiki to issues@metron.incubator.
>>  apache.org
>>  I don't have access to create/assign JIRAs to myself
>>  I don't have visibility/access to certain JIRA featuers
>>  I can't create or view a wiki entry
>>  etc
>>
>>  ---
>>  Thank you,
>>
>>  James Sirota
>>  PPMC- Apache Metron (Incubating)
>>  jsirota AT apache DOT org
>
> --
> Nick Allen 

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Re: [VOTE] Release Process

[Casey Stella]

+1 (binding)

On Tue, Jan 17, 2017 at 11:17 PM, James Sirota  wrote:

> I made the revisions based on the discuss thread
>
> The document is available here:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66854770
>
> And is also attached for reference to this email.
>
> Please vote +1, -1, or 0 for neutral.  The vote will last 72 hours.
>
> Thanks,
> James
>
> -
> Metron Release Types
> There are two types of Metron releases:
> Feature Release (FR) - this is a release that has a significant step
> forward in feature capability and is denoted by an upgrade of the second
> digit
> Maintenance Release (MR) - this is a set of patches and fixes that are
> issued following the FR and is denoted by an upgrade of the third digit
> Release Naming Convention
> Metron build naming convention is as follows: 0.[FR].[MR].  We keep the 0.
> notation to signify that the project is still under active development and
> we will hold a community vote to go to 1.x at a future time
> Initiating a New Metron Release
> Immediately upon the release of the previous Metron release create two
> branches: FR ++ and MR.  Create the FR++ branch by incrementing the second
> digit like so 0.[FR++].0.  Create the MR branch for the previous Metron
> release by incrementing the second digit of the previous release like so
> 0.[FR].[MR].  All patches to the previous Metron release will be checked in
> under the MR branch and where it makes sense also under the FR branch.  All
> new features will be checked in under the FR branch.
> Creating a Feature Release
> Step 1 - Initiate a discuss thread
> Prior to the release The Release manager should do the following
> (preferably a month before the release):
> Make sure that the list of JIRAs slated for the release accurately
> reflects to reflects the pull requests that are currently in master
> Construct an email to the Metron dev board (dev@metron.incubator.apache.
> org) which discusses with the community the desire to do a release. This
> email should contain the following:
> The list of JIRAs slated for the release with descriptions (use the output
> of git log and remove all the JIRAs from the last release’s changelog)
> A solicitation of JIRAs that should be included with the next release.
> Users should rate them as must/need/good to have as well as volunteering.
> A release email template is provided here.
> Step 2 - Monitor and Verify JIRAs
> Once the community votes for additional JIRAs they want included in the
> release verify that the pull requests are in before the release, close
> these JIRAs and tag them with the release name. All pull requests and JIRAs
> that were not slated for this release will go into the next releases.  The
> release manager should continue to monitor the JIRA to ensure that the
> timetable is on track until the release date.  On the release date the
> release manager should message the Metron dev board (
> dev@metron.incubator.apache.org) announcing the code freeze for the
> release.
> Step 3 - Create the Release Branch and Increment Metron version
> Create an branch for the release (from a repo cloned from
> https://git-wip-us.apache.org/repos/asf/incubator-metron.git). (assuming
> the release is 0.[FR++].0 and working from master):
> git checkout -b Metron_0.[FR++].0
> git push --set-upstream origin Metron_0.[FR++].0
> File a JIRA to increment the Metron version to 0.[FR++].0.  Either do it
> yourself or have a community member increment the build version for you.
> You can look at a pull request for a previous build to see how this is
> done.   METRON-533 - Up the version for release DONE
> Also, the release manager should have a couple of things set up:
> A SVN clone of the repo at https://dist.apache.org/repos/
> dist/dev/incubator/metron, We will refer to this as the dev repo.  It
> will hold the release candidate artifacts
> A SVN clone of the repo at https://dist.apache.org/repos/
> dist/release/incubator/metron, We will refer to this as the release
> repo.  It will hold the release artifacts.
> Step 4 - Create the Release Candidate
>
> Now, for each release candidate, we will tag from that branch. Assuming
> that this is RC1:
> git checkout Metron_0.[FR++].0 && git pull
> git tag apache-metron-0.[FR++].0-rc1-incubating
> git push origin —tags
> Now we must create the release candidate tarball. From the apache repo,
> you should run:
>
>  git archive --prefix=apache-metron-0.[FR++].0-rc1-incubating/
>  apache-metron-0.[FR++].0-rc1-incubating | gzip >
>  apache-metron-0.[FR++].0-rc-incubating.tar.gz
>
> We will refer to this as the release candidate tarball. *Note: Per Apache
> policy, the hardware used to create the candidate tarball must be owned by
> the release manager.
> The artifacts for a release (or a release candidate, for that matter) are
> as follows:
> Release (candidate) Tarball
>  MD5 hash of the release tarball (md5 apache-metron-Now, we must grab the
> release candidate binary 

[GitHub] incubator-metron pull request #316: METRON-503: Metron REST API

[merrimanr]

Github user merrimanr closed the pull request at:

https://github.com/apache/incubator-metron/pull/316


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #419: METRON-664: Make the index configuration...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/incubator-metron/pull/419

METRON-664: Make the index configuration per-writer with enabled/disabled

Currently the index configuration is per-sensor and the properties 
specified are identical for every writer.  Also, the ability to turn off a 
given writer for a given sensor is not available.

This JIRA seeks to remedy that by:
* Making the per-sensor indexing config have per-writer sections for the 
properties available to configure
* Adding a new per-writer property `enabled` to indicate whether the writer 
is turned on or off (default on).

Please see the `metron-indexing` documentation for examples of the new 
configs.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron METRON-664

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/419.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #419


commit 5ee9295657a218235f4f38d5475693bdebab44f3
Author: cstella 
Date:   2017-01-17T00:12:59Z

First cut.

commit 58af93ea50f1ac20ef61e232e125becaf4756a29
Author: cstella 
Date:   2017-01-17T16:21:56Z

Updating to add warning message.

commit 46f806061e0a64da78c3539cec418abb78978875
Author: cstella 
Date:   2017-01-17T21:54:18Z

Updating stellar management functions

commit 2c3d8dcd0b6c8a537a1d8439483f794cdd7400ca
Author: cstella 
Date:   2017-01-18T14:07:59Z

Updated readme.

commit d0eea5db097701aff83e9df5dcee87da2b0724d8
Author: cstella 
Date:   2017-01-18T14:17:53Z

Updating indexing functions and docs.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Release Process

[Otto Fowler]

Maybe this can be scripted as well


On January 17, 2017 at 15:34:10, Casey Stella (ceste...@gmail.com) wrote:

Larry,

Thanks for the info. In that case, then the following passage:

> Now, we must grab the release candidate binary from the github releases
> page (https://github.com/apache/incubator-metron/releases). In our case,
> for RC1, that would be
>
https://github.com/apache/incubator-metron/archive/apache-metron-0.[FR++].0-rc1-incubating.tar.gz
We
> will refer to this as the release candidate tarball.


Should be replaced with:

> Now we must create the release candidate tarball. From the apache repo,
> you should run:
> git archive --prefix=apache-metron-0.[FR++].0-rc1-incubating/
> apache-metron-0.[FR++].0-rc1-incubating | gzip >
> apache-metron-0.[FR++].0-rc-incubating.tar.gz We will refer to this as
the
> release candidate tarball.


On Tue, Jan 17, 2017 at 3:20 PM, larry mccay  wrote:

> It is technically a violation of apache release policy to build releases
in
> such a way [1]:
>
> MUST RELEASES BE BUILT ON HARDWARE OWNED AND CONTROLLED BY THE COMMITTER?
> 
>
> Strictly speaking, releases must be verified
>  releases/compare_dirs.pl>
> on
> hardware owned and controlled by the committer. That means hardware the
> committer has physical possession and control of and exclusively full
> administrative/superuser access to. That's because only such hardware is
> qualified to hold a PGP private key, and the release should be verified
on
> the machine the private key lives on or on a machine as trusted as that.
>
> Practically speaking, when a release consists of anything beyond an
archive
> (e.g., tarball or zip file) of a source control tag, the only practical
way
> to validate that archive is to build it locally; manually inspecting
> generated files (especially binary files) is not feasible. So, basically,
> "Yes".
>
> *Note: This answer refers to the process used to produce a release
artifact
> from a source control tag. It does not refer to testing that artifact for
> technical quality.*
>
>
> Knox is still using the archive from a jenkins build and is also out of
> compliance.
>
> We will need to eventually change this approach as well.
>
> The threat is that someone could compromise such a remote system in a way
> that adds additional classes or alters the code in someway that the
project
> will then be propagating this compromised binary under the Apache brand.
>
>
> 1. http://www.apache.org/dev/release.html#owned-controlled-hardware
>
> On Tue, Jan 17, 2017 at 2:43 PM, Casey Stella  wrote:
>
> > Hey Matt,
> >
> > Github actually constructs the tarball for us using git archive (for
> every
> > tag, for that matter). We don't explicitly push any tarball to github.
> > The reason why we pull the tarball from github directly is that we ship
a
> > .gitattributes to define what gets put in the tarball. (we don't ship
the
> > site for instance). This was just easier to describe than to walk
> through
> > using git archive. I don't think it's hurting anything necessarily, but
> I
> > might be wrong.
> >
> > Regarding Step 7, that can be made explicit, but the link is part of
the
> > VOTE template.
> >
> > Regarding maintenance releases, 1 and 2 may be skipped for a
maintenance
> > release, but the rest really should be followed. It's a release and
must
> > abide by apache requirements, I think. Maybe the mentors could chime
in,
> > though.
> >
> > Regarding Security break-fix, I'm not sure. Perhaps the mentors can
> chime
> > in?
> >
> > Casey
> >
> > On Mon, Jan 16, 2017 at 4:05 PM, Matt Foley  wrote:
> >
> > > Overall, a great contribution. I suspect that as the next couple
> Release
> > > Managers go thru it, they’ll find various glitches to clean up, but
> > that’s
> > > fine.
> > > Bravo especially for the last couple paragraphs (Ensuring Consistency
> > > between Feature and Maint Releases), which are very good.
> > >
> > > One major issue: Step 4 says:
> > > >> Now, we must grab the release candidate binary from the github
> > releases
> > > page (https://github.com/apache/incubator-metron/releases).
> > >
> > > Missing step! How did the tarball get there?
> > >
> > > Also, I don’t think the tarball should be first pushed to github.
What
> > > benefit does this provide, vs just pushing directly to the dev repo (
> > > https://dist.apache.org/repos/dist/dev/incubator/metron )?
> > >
> > > Step 7 should state that the call for vote will include a link to the
> RC
> > > release in the dev repo.
> > >
> > > >>Creating a Maintenance Release
> > > >> … if a critical JIRA comes in that requires an immediate patch we
> may
> > > forego steps 2-5 …
> > >
> > > Eh? I can see skipping steps 1 and 2, and abbreviating steps 5 and 6,
> > but
> > > steps 3 and 4 are purely mechanical and seem needed by definition to
> > make 

Re: [PROPOSAL] Reduce Reliance on Ansible for Deployment

[Otto Fowler]

What does this do for Monit?


On January 17, 2017 at 10:34:32, David Lyle (dlyle65...@gmail.com) wrote:

In our "Dev Guide and Committer Review Guide additions" discussion, we had
a bit of a side discussion about reducing reliance (perhaps to zero) on
Ansible for our installation.

It seemed there was consensus around that idea (if not, please let me
know), so I propose the following steps to get there:

1) Refactor existing Ansible deployment to use the Ambari MPack to install
metron-common, metron-enrichments and metron-parsers.
2) Regenerate quick-dev to leverage the change.
3) Create rpm packages for all deployed components that don't currently
have them.
- Sensor probes
- Sensor stubs
4) Create MPack service defs for the RPMs in (2).
5) Refactor existing Ansible deployment to use the Ambari MPack to install
all services.
6) Regenerate quick-dev to leverage the change.
7) Plan iteration 2 to see if there are other opportunities to reduce our
use of Ansible.

One note: if we decide to go this direction, it'd be helpful if, during the
transition, we stopped adding additional Ansible deployment code.

Thoughts?

Thanks,

-David...

Re: [VOTE] Reporting Issues Wiki

[Nick Allen]

+1 binding

On Thu, Jan 5, 2017 at 5:41 PM, James Sirota  wrote:

> Based on feedback from the discuss thread.
>
> Please vote +1, -1, or 0.  The vote will be open for 72 hours
>
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=67635199
>
>
> All “I have found a bug” issues are considered developer-level issues.
> Please report all developer-level issues to dev@metron.incubator.apache.
> org.  Examples of developer issues would be:
> Project fails to compile or is failing unit or integration tests
> Project or individual components fail to install
> There are error messages or failures in my logs
> I need help with coding or extending a specific component
> etc...
> After discussion of the issue on the JIRA if it is clear that you found a
> bug then you should file a JIRA (unless you found a security
> vulnerability).  Follow up on the mailing lists if you want advice with
> respect to workaround or a local fix.  Our JIRA is located here.
>
> All  “I have a problem” or "How do you use x" issues are usability
> issues.  If you are an end-user of the product and have a comment or
> question then use u...@metron.incubator.apache.org.  If you have a
> problem and a strong suspicion that you might have found a bug, please
> cross-reference dev@metron.incubator.apache.org as well
> I don't understand the UI, what does button x do?
> What should the output of function x be?
> It would be nice if I had feature x along with feature y
> etc...
>
> If you found a security-related issue, please report immediately to
> secur...@metron.incubator.apache.org. Please adhere to the following
> Apache policy found here. DO NOT FILE A JIRA, DO NOT POST ON ANY OTHER BOARD
> I can get access to data I should not have access to
> I have privileges to do things I should not be allowed to do
> I found that this project is susceptible to an exploit
> etc...
>
> Please report issues related to the JIRA/Wiki to issues@metron.incubator.
> apache.org
> I don't have access to create/assign JIRAs to myself
> I don't have visibility/access to certain JIRA featuers
> I can't create or view a wiki entry
> etc
>
>
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>



-- 
Nick Allen 

Re: [VOTE] Reporting Issues Wiki

[zeo...@gmail.com]

+1 (non-binding)

On Wed, Jan 18, 2017, 5:53 AM Anand Subramanian <
asubraman...@hortonworks.com> wrote:

> +1 (non-binding)
>
>
>
>
> On 1/18/17, 10:20 AM, "James Sirota"  wrote:
>
> >Can I get some more votes on this so I can close it out?
> >
> >10.01.2017, 13:50, "Ryan Merriman" :
> >> +1 (binding)
> >>
> >> On Mon, Jan 9, 2017 at 8:23 AM, Casey Stella 
> wrote:
> >>
> >>>  +1 (binding)
> >>>
> >>>  On Fri, Jan 6, 2017 at 7:43 PM, Kyle Richardson <
> kylerichards...@gmail.com
> >>>  >
> >>>  wrote:
> >>>
> >>>  > 0 (binding)
> >>>  >
> >>>  > I think it's good but it just feels a little cumbersome still.
> >>>  >
> >>>  > -Kyle
> >>>  >
> >>>  > > On Jan 6, 2017, at 7:53 AM, JJ Meyer  wrote:
> >>>  > >
> >>>  > > +1 (non-binding)
> >>>  > >
> >>>  > > What do you think about changing `*DO NOT FILE A JIRA, DO NOT
> POST ON
> >>>  ANY
> >>>  > > OTHER BOARD` *to standard case, but use a confluence warning
> macro? The
> >>>  > all
> >>>  > > caps just makes me feel like I'm being yelled at :)
> >>>  > >
> >>>  > >> On Thu, Jan 5, 2017 at 8:10 PM, zeo...@gmail.com <
> zeo...@gmail.com>
> >>>  > wrote:
> >>>  > >>
> >>>  > >> +1 (non-binding)
> >>>  > >>
> >>>  > >>> On Thu, Jan 5, 2017, 8:30 PM Matt Foley 
> wrote:
> >>>  > >>>
> >>>  > >>> +1 (non-binding)
> >>>  > >>>
> >>>  > >>> One typo is still in there:
> >>>  > > After discussion of the issue on the JIRA if it is clear that
> you
> >>>  > >>> found a bug then you should file a JIRA
> >>>  > >>> should be
> >>>  > > After discussion of the issue on the mailing list if it is
> clear
> >>>  > >>> that you found a bug then you should file a JIRA
> >>>  > >>>
> >>>  > >>> Cheers,
> >>>  > >>> --Matt
> >>>  > >>>
> >>>  > >>>
> >>>  > >>> On 1/5/17, 2:41 PM, "James Sirota"  wrote:
> >>>  > >>>
> >>>  > >>> Based on feedback from the discuss thread.
> >>>  > >>>
> >>>  > >>> Please vote +1, -1, or 0. The vote will be open for 72 hours
> >>>  > >>>
> >>>  > >>>
> >>>  > >>>
> >>>  > >>> https://cwiki.apache.org/confluence/pages/viewpage.
> >>>  > >> action?pageId=67635199
> >>>  > >>>
> >>>  > >>>
> >>>  > >>> All “I have found a bug” issues are considered developer-level
> >>>  > >>> issues. Please report all developer-level issues to
> >>>  > >>> dev@metron.incubator.apache.org. Examples of developer issues
> would
> >>>  > be:
> >>>  > >>> Project fails to compile or is failing unit or integration tests
> >>>  > >>> Project or individual components fail to install
> >>>  > >>> There are error messages or failures in my logs
> >>>  > >>> I need help with coding or extending a specific component
> >>>  > >>> etc...
> >>>  > >>> After discussion of the issue on the JIRA if it is clear that
> you
> >>>  > >>> found a bug then you should file a JIRA (unless you found a
> security
> >>>  > >>> vulnerability). Follow up on the mailing lists if you want
> advice
> >>>  with
> >>>  > >>> respect to workaround or a local fix. Our JIRA is located here.
> >>>  > >>>
> >>>  > >>> All “I have a problem” or "How do you use x" issues are
> usability
> >>>  > >>> issues. If you are an end-user of the product and have a
> comment or
> >>>  > >>> question then use u...@metron.incubator.apache.org. If you
> have a
> >>>  > >>> problem and a strong suspicion that you might have found a bug,
> >>>  please
> >>>  > >>> cross-reference dev@metron.incubator.apache.org as well
> >>>  > >>> I don't understand the UI, what does button x do?
> >>>  > >>> What should the output of function x be?
> >>>  > >>> It would be nice if I had feature x along with feature y
> >>>  > >>> etc...
> >>>  > >>>
> >>>  > >>> If you found a security-related issue, please report immediately
> >>>  to
> >>>  > >>> secur...@metron.incubator.apache.org. Please adhere to the
> following
> >>>  > >>> Apache policy found here. DO NOT FILE A JIRA, DO NOT POST ON ANY
> >>>  OTHER
> >>>  > >> BOARD
> >>>  > >>> I can get access to data I should not have access to
> >>>  > >>> I have privileges to do things I should not be allowed to do
> >>>  > >>> I found that this project is susceptible to an exploit
> >>>  > >>> etc...
> >>>  > >>>
> >>>  > >>> Please report issues related to the JIRA/Wiki to
> >>>  > >>> iss...@metron.incubator.apache.org
> >>>  > >>> I don't have access to create/assign JIRAs to myself
> >>>  > >>> I don't have visibility/access to certain JIRA featuers
> >>>  > >>> I can't create or view a wiki entry
> >>>  > >>> etc
> >>>  > >>>
> >>>  > >>>
> >>>  > >>> ---
> >>>  > >>> Thank you,
> >>>  > >>>
> >>>  > >>> James Sirota
> >>>  > >>> PPMC- Apache Metron (Incubating)
> >>>  > >>> jsirota AT apache DOT org
> >>>  > >>>
> >>>  > >>>
> >>>  > >>>
> >>>  > >>>
> >>>  > >>> --
> >>>  > >>
> >>>  > >> Jon
> >>>  > >>
> >>>  > >> Sent from my mobile device
> >>>  > >>
> >>>  >
> >
> >---
> >Thank you,
> >
> >James Sirota
> >PPMC- 

Re: [VOTE] Reporting Issues Wiki

[Anand Subramanian]

+1 (non-binding)




On 1/18/17, 10:20 AM, "James Sirota"  wrote:

>Can I get some more votes on this so I can close it out?
>
>10.01.2017, 13:50, "Ryan Merriman" :
>> +1 (binding)
>>
>> On Mon, Jan 9, 2017 at 8:23 AM, Casey Stella  wrote:
>>
>>>  +1 (binding)
>>>
>>>  On Fri, Jan 6, 2017 at 7:43 PM, Kyle Richardson >>  >
>>>  wrote:
>>>
>>>  > 0 (binding)
>>>  >
>>>  > I think it's good but it just feels a little cumbersome still.
>>>  >
>>>  > -Kyle
>>>  >
>>>  > > On Jan 6, 2017, at 7:53 AM, JJ Meyer  wrote:
>>>  > >
>>>  > > +1 (non-binding)
>>>  > >
>>>  > > What do you think about changing `*DO NOT FILE A JIRA, DO NOT POST ON
>>>  ANY
>>>  > > OTHER BOARD` *to standard case, but use a confluence warning macro? The
>>>  > all
>>>  > > caps just makes me feel like I'm being yelled at :)
>>>  > >
>>>  > >> On Thu, Jan 5, 2017 at 8:10 PM, zeo...@gmail.com 
>>>  > wrote:
>>>  > >>
>>>  > >> +1 (non-binding)
>>>  > >>
>>>  > >>> On Thu, Jan 5, 2017, 8:30 PM Matt Foley  wrote:
>>>  > >>>
>>>  > >>> +1 (non-binding)
>>>  > >>>
>>>  > >>> One typo is still in there:
>>>  > > After discussion of the issue on the JIRA if it is clear that you
>>>  > >>> found a bug then you should file a JIRA
>>>  > >>> should be
>>>  > > After discussion of the issue on the mailing list if it is clear
>>>  > >>> that you found a bug then you should file a JIRA
>>>  > >>>
>>>  > >>> Cheers,
>>>  > >>> --Matt
>>>  > >>>
>>>  > >>>
>>>  > >>> On 1/5/17, 2:41 PM, "James Sirota"  wrote:
>>>  > >>>
>>>  > >>> Based on feedback from the discuss thread.
>>>  > >>>
>>>  > >>> Please vote +1, -1, or 0. The vote will be open for 72 hours
>>>  > >>>
>>>  > >>>
>>>  > >>>
>>>  > >>> https://cwiki.apache.org/confluence/pages/viewpage.
>>>  > >> action?pageId=67635199
>>>  > >>>
>>>  > >>>
>>>  > >>> All “I have found a bug” issues are considered developer-level
>>>  > >>> issues. Please report all developer-level issues to
>>>  > >>> dev@metron.incubator.apache.org. Examples of developer issues would
>>>  > be:
>>>  > >>> Project fails to compile or is failing unit or integration tests
>>>  > >>> Project or individual components fail to install
>>>  > >>> There are error messages or failures in my logs
>>>  > >>> I need help with coding or extending a specific component
>>>  > >>> etc...
>>>  > >>> After discussion of the issue on the JIRA if it is clear that you
>>>  > >>> found a bug then you should file a JIRA (unless you found a security
>>>  > >>> vulnerability). Follow up on the mailing lists if you want advice
>>>  with
>>>  > >>> respect to workaround or a local fix. Our JIRA is located here.
>>>  > >>>
>>>  > >>> All “I have a problem” or "How do you use x" issues are usability
>>>  > >>> issues. If you are an end-user of the product and have a comment or
>>>  > >>> question then use u...@metron.incubator.apache.org. If you have a
>>>  > >>> problem and a strong suspicion that you might have found a bug,
>>>  please
>>>  > >>> cross-reference dev@metron.incubator.apache.org as well
>>>  > >>> I don't understand the UI, what does button x do?
>>>  > >>> What should the output of function x be?
>>>  > >>> It would be nice if I had feature x along with feature y
>>>  > >>> etc...
>>>  > >>>
>>>  > >>> If you found a security-related issue, please report immediately
>>>  to
>>>  > >>> secur...@metron.incubator.apache.org. Please adhere to the following
>>>  > >>> Apache policy found here. DO NOT FILE A JIRA, DO NOT POST ON ANY
>>>  OTHER
>>>  > >> BOARD
>>>  > >>> I can get access to data I should not have access to
>>>  > >>> I have privileges to do things I should not be allowed to do
>>>  > >>> I found that this project is susceptible to an exploit
>>>  > >>> etc...
>>>  > >>>
>>>  > >>> Please report issues related to the JIRA/Wiki to
>>>  > >>> iss...@metron.incubator.apache.org
>>>  > >>> I don't have access to create/assign JIRAs to myself
>>>  > >>> I don't have visibility/access to certain JIRA featuers
>>>  > >>> I can't create or view a wiki entry
>>>  > >>> etc
>>>  > >>>
>>>  > >>>
>>>  > >>> ---
>>>  > >>> Thank you,
>>>  > >>>
>>>  > >>> James Sirota
>>>  > >>> PPMC- Apache Metron (Incubating)
>>>  > >>> jsirota AT apache DOT org
>>>  > >>>
>>>  > >>>
>>>  > >>>
>>>  > >>>
>>>  > >>> --
>>>  > >>
>>>  > >> Jon
>>>  > >>
>>>  > >> Sent from my mobile device
>>>  > >>
>>>  >
>
>--- 
>Thank you,
>
>James Sirota
>PPMC- Apache Metron (Incubating)
>jsirota AT apache DOT org
>