date:20170104

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/399#discussion_r94701887
  
--- Diff: site/documentation/index.md ---
@@ -69,3 +70,29 @@ title: Apache Metron (Incubating) Documentation
 https://cwiki.apache.org/confluence/display/METRON/Documentation; 
target="_blank">LEARN MORE
 
 
+
+
+
+
+Current Release
+
+
+
+
+http://www.apache.org/dyn/closer.cgi/incubator/metron/0.3.0/apache-metron-0.3.0-incubating.tar.gz;
 target="new">
+apache-metron-0.3.0-incubating.tar.gz
+
+ [ http://www.apache.org/dyn/closer.cgi/incubator/metron/0.3.0/apache-metron-0.3.0-incubating.tar.gz.asc;
 target="new">
+PGP
+ ]
+ [ http://www.apache.org/dyn/closer.cgi/incubator/metron/0.3.0/apache-metron-0.3.0-incubating.tar.gz.sha;
 target="new">
+SHA512
+ ]
+ [ http://www.apache.org/dyn/closer.cgi/incubator/metron/0.3.0/apache-metron-0.3.0-incubating.tar.gz.md5;
 target="new">
+MD5
+  ]
--- End diff --

Sorry, but as stated in the Jira:
- All links to the downloadable distribution artifacts MUST NOT reference 
the main Apache dist web site (dist.apache.org). Instead, they should use the 
standard scripting mechanisms to distribute the load between the mirror sites 
(see http://www.apache.org/dyn/closer.cgi/ ).
- All links to checksums, detached signatures and public keys MUST 
reference the main Apache dist web site (dist.apache.org), and use https (SSL).

Therefore, this section should read:
```
 +http://www.apache.org/dyn/closer.cgi/incubator/metron/0.3.0/apache-metron-0.3.0-incubating.tar.gz;
 target="new">
 +apache-metron-0.3.0-incubating.tar.gz
 +
 + [ https://dist.apache.org/repos/dist/release/incubator/metron/0.3.0/apache-metron-0.3.0-incubating.tar.gz.asc;
 target="new">
 +PGP
 + ]
 + [ https://dist.apache.org/repos/dist/release/incubator/metron/0.3.0/apache-metron-0.3.0-incubating.tar.gz.sha;
 target="new">
 +SHA512
 + ]
 + [ https://dist.apache.org/repos/dist/release/incubator/metron/0.3.0/apache-metron-0.3.0-incubating.tar.gz.md5;
 target="new">
 +MD5
 +  ]
```
The first item (the tarball) is correct, the other three must use https and 
point at the dist.apache.org site.
Thanks for doing this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Release Process

2017-01-04 Thread Matt Foley

Hi James, is there a formatted version of this somewhere we can look at?
Thanks,
--Matt

On 1/4/17, 1:53 PM, "James Sirota" wrote:

Revised as per additional comments. Are there more comments? Or can we
put this up for a vote?

https://git-wip-us.apache.org/repos/asf?p=incubator-metron.git;a=shortlog;h=refs/tags/apache-metron-0.3.0-rc1-incubating
Step 10 - Stage the release
The next thing to do is to sign and stage the release including the
DISCLAIMER, KEYS, and LICENSE files. A properly signed and staged release can
be found here:

https://dist.apache.org/repos/dist/dev/incubator/metron/0.3.0-RC1-incubating/
* Make sure you have your correct profile and keys uploaded to
https://id.apache.org/ to properly sign the release and to get access to
dist.apache.org
Step 11 - Call for a community release vote
Next initiate a [VOTE] threat on the dev list to announce the build vote.
The vote email template can be found here: Build Vote Template. Allow at least
72 hours for the community to vote on the release. When you get enough votes
close the vote by replying [RESULT][VOTE] to the email thread with the tally of
all the votes
Step 12 - Call for a incubator release vote
Upon successful completion of step 11, repeat, but now send the email to
the incubator general boards. The email should be identical. Again, wait for
at least 72 hours and then close the vote.
Step 13 - Stage the finished release

Re: [DISCUSS] Coding Guidelines

Does anyone else has any comments or objections to the document?  Or can we put 
this up for a vote?

21.12.2016, 13:24, "Kyle Richardson" :
> +1 Thanks for the updates to the documentation and testing sections.
>
> -Kyle
>
> On Wed, Dec 21, 2016 at 3:17 PM, Otto Fowler 
> wrote:
>
>>  +1
>>
>>  On December 21, 2016 at 14:56:06, Michael Miklavcic (
>>  michael.miklav...@gmail.com) wrote:
>>
>>  Works for me also.
>>
>>  On Wed, Dec 21, 2016 at 12:38 PM, Matt Foley  wrote:
>>
>>  > Works for me, thanks.
>>  >
>>  > On 12/21/16, 11:21 AM, "Casey Stella"  wrote:
>>  >
>>  > Sure, how about making it generic to "a deployed cluster"?
>>  >
>>  > On Wed, Dec 21, 2016 at 2:20 PM, Matt Foley  wrote:
>>  >
>>  > > +1 on Casey’s first edit. However, wrt the second, can we please not
>>  > > require vagrant? Any of our single-node test deployments, including
>>  > > vagrant, ansible, mpack, or (soon :-) docker, should be acceptable.
>>  > >
>>  > > Thanks,
>>  > > --Matt (who can’t run vagrant workably on the systems available to
>>  > me)
>>  > >
>>  > >
>>  > > On 12/21/16, 8:52 AM, "Michael Miklavcic" <
>>  > michael.miklav...@gmail.com>
>>  > > wrote:
>>  > >
>>  > > Agreed on Casey's addition to 2.5. What do you think about
>>  > saying the
>>  > > pla
>>  > > should be stated on the PR, since that will be replicated to Jira
>>  > > automatically?
>>  > >
>>  > > On Wed, Dec 21, 2016 at 7:49 AM, Casey Stella <
>>  > ceste...@gmail.com>
>>  > > wrote:
>>  > >
>>  > > > Oh, one more, I propose the following addition to 2.5:
>>  > > > >
>>  > > > > JIRAs will have a description of how to exercise the
>>  > functionality
>>  > > in a
>>  > > > > step-by-step manner on a Quickdev vagrant instance to aid
>>  > review
>>  > > and
>>  > > > > validation.
>>  > > >
>>  > > >
>>  > > > When Mike, Otto and I moved the system to the current version
>>  > of
>>  > > Storm, we
>>  > > > needed a broader smoke test than just running data through that
>>  > > exercised a
>>  > > > variety of the features. We pulled those smoke tests from the
>>  > various
>>  > > > discussions in the JIRAs.
>>  > > >
>>  > > >
>>  > > >
>>  > > > On Wed, Dec 21, 2016 at 9:38 AM, Casey Stella <
>>  > ceste...@gmail.com>
>>  > > wrote:
>>  > > >
>>  > > > > We have been having a lively discussion on METRON-590 (see
>>  > > > > https://github.com/apache/incubator-metron/pull/395) around
>>  > > creating
>>  > > > > multiple abstractions to do the same (or very nearly the
>>  > same)
>>  > > thing.
>>  > > > >
>>  > > > > I'd like to propose an addition to section 2.3 which reads:
>>  > > > >
>>  > > > >> Contributions which provide abstractions which are either
>>  > very
>>  > > similar
>>  > > > to
>>  > > > >> or a subset of existing abstractions should use and extend
>>  > > existing
>>  > > > >> abstractions rather than provide competing abstractions
>>  > unless
>>  > > > engineering
>>  > > > >> exigencies (e.g. performance ) make such an operation
>>  > impossible
>>  > > without
>>  > > > >> compromising core functionality of the platform.
>>  > > > >
>>  > > > >
>>  > > > > I'd like to suggest the following anecdote from the early
>>  > years of
>>  > > the
>>  > > > > codebase to justify the above:
>>  > > > >
>>  > > > > Stellar started as a predicate language only for threat
>>  > triage
>>  > > rules. As
>>  > > > > such, when the task of creating Field Transformations came
>>  > to me, I
>>  > > > needed
>>  > > > > something like Stellar except I needed it to return arbitrary
>>  > > objects,
>>  > > > > rather than just booleans. In my infinite wisdom, I chose to
>>  > fork
>>  > > the
>>  > > > > language, create a second, more specific DSL for field
>>  > > transformations,
>>  > > > > thereby creating "Metron Query Language" and "Metron
>>  > Transformation
>>  > > > > Language."
>>  > > > >
>>  > > > > I felt a nagging feeling at the time that I should just
>>  > expand the
>>  > > query
>>  > > > > language, but I convinced myself that it would require too
>>  > much
>>  > > testing
>>  > > > and
>>  > > > > it would be a change that was too broad in scope. It took 3
>>  > months
>>  > > for me
>>  > > > > to get around to unifying those languages and if we had more
>>  > > people using
>>  > > > > it, it would have been an absolute nightmare.
>>  > > > >
>>  > > > > On Wed, Dec 21, 2016 at 9:31 AM, Casey Stella <
>>  > ceste...@gmail.com>
>>  > > > wrote:
>>  > > > >
>>  > > > >> Yeah, I +1 the notion of thorough automated tests.
>>  > > > >>
>>  > > > >> On Tue, Dec 20, 2016 at 4:36 PM, Matt Foley <
>>  > ma...@apache.org>
>>  > > wrote:
>>  > > > >>
>>  > > > >>> Hard to mark diffs in text-only mode :-) My proposed
>>  > change is:
>>  > > > >>>
>>  > > > >>> >> All merged patches will be reviewed with the
>>  > expectation that
>>  > > > >>> thorough automated tests shall be provided and are
>>  > consistent
>>  >

Re: [DISCUSS] Release Process

Revised as per additional comments. Are there more comments? Or can we put
this up for a vote?

Release Process [DRAFT]
Skip to end of metadata
Created by James Sirota, last modified just a moment ago Go to start of metadata
Metron Release Types
There are two types of Metron releases:
Feature Release (FR) - this is a release that has a significant step forward in
feature capability and is denoted by an upgrade of the second digit
Maintenance Release (MR) - this is a set of patches and fixes that are issued
following the FR and is denoted by an upgrade of the third digit
Release Naming Convention
Metron build naming convention is as follows: 0.[FR].[MR]. We keep the 0.
notation to signify that the project is still under active development and we
will hold a community vote to go to 1.x at a future time
Initiating a New Metron Release
Immediately upon the release of the previous Metron release create two
branches: FR ++ and MR. Create the FR++ branch by incrementing the second
digit like so 0.[FR++].0. Create the MR branch for the previous Metron release
by incrementing the second digit of the previous release like so 0.[FR].[MR].
All patches to the previous Metron release will be checked in under the MR
branch and where it makes sense also under the FR branch. All new features
will be checked in under the FR branch.
Creating a Feature Release
Step 1 - Initiate a discuss thread
A week before a new feature release initiate a discuss thread on the Metron dev
board announcing the upcoming release and asking the community which still
outstanding pull requests people want to include in the next build.
Step 2 - Verify JIRA
Go through the JIRA and verify that all pull requests that were merged for the
upcoming build have JIRAs that are in a closed state and are appropriately
labelled with the next build version.
Step 3 - Announce a code freeze
A day before the release date comment on the discuss thread and let people know
that the release is ready. Go through the JIRAs for pull requests that came in
during the last week and make sure they are labelled with the next build
version.
Step 4 - Increment Metron version
File a JIRA to increment the Metron version to 0.[FR++].0. Either do it
yourself or have a community member increment the build version for you. You
can look at a pull request for a previous build to see how this is done
Step 5 - Increment build version
File a JIRA to increment the Metron version to 0.[FR++].0-RC(n), where RC(n) is
the number of the release candidate. Sometimes mistakes occur (builds may get
voted down) so it will take multiple RCs to get a build through the vote. The
RC(n) will be removed after the successful vote.
Step 6 - Verify the build
Go through the build verification checklist to verify that everything works.
These instructions can be found here: Verifying Builds
Step 7 - Verify licensing
Make sure the release compiles with the following Apache licensing guidelines:
http://www.apache.org/foundation/license-faq.html
Step 8 - Generate the changes file
Go through the JIRA to generate the changes file, which contains a list of all
JIRAs included in the upcoming release. An example of a changes file can be
found here:
https://dist.apache.org/repos/dist/dev/incubator/metron/0.3.0-RC1-incubating/CHANGES
Step 9 - Tag the RC release
Tag the release for the RC in case we need to roll back at some point. An
example of a valid tag can be seen here:
https://git-wip-us.apache.org/repos/asf?p=incubator-metron.git;a=shortlog;h=refs/tags/apache-metron-0.3.0-rc1-incubating
Step 10 - Stage the release
The next thing to do is to sign and stage the release including the DISCLAIMER,
KEYS, and LICENSE files. A properly signed and staged release can be found
here:
https://dist.apache.org/repos/dist/dev/incubator/metron/0.3.0-RC1-incubating/
* Make sure you have your correct profile and keys uploaded to
https://id.apache.org/ to properly sign the release and to get access to
dist.apache.org
Step 11 - Call for a community release vote
Next initiate a [VOTE] threat on the dev list to announce the build vote. The
vote email template can be found here: Build Vote Template. Allow at least 72
hours for the community to vote on the release. When you get enough votes
close the vote by replying [RESULT][VOTE] to the email thread with the tally of
all the votes
Step 12 - Call for a incubator release vote
Upon successful completion of step 11, repeat, but now send the email to the
incubator general boards. The email should be identical. Again, wait for at
least 72 hours and then close the vote.
Step 13 - Stage the finished release
If the vote fails at any stage then incorporate feedback, create another RC,
and repeat. If both votes pass then stage the resulting artifacts here:
https://dist.apache.org/repos/dist/release/incubator/metron/
Step 14 - Announce build
Send a discuss thread to the Metron dev boards announcing the new Metron build
Creating

Re: [DISCUSS] Reporting Metron Issues

Updated as per comments.  Are there more changes?  Or can we put this up for a 
vote?

All “I have found a bug” issues are considered developer-level issues.  Please 
report all developer-level issues to dev@metron.incubator.apache.org.  Examples 
of developer issues would be:
Project fails to compile or is failing unit or integration tests
Project or individual components fail to install 
There are error messages or failures in my logs
I need help with coding or extending a specific component
etc...
After discussion of the issue on the JIRA if it is clear that you found a bug 
then you should file a JIRA (unless you found a security vulnerability).  
Follow up on the mailing lists if you want advice with respect to workaround or 
a local fix.  Our JIRA is located here. 

All  “I have a problem” or "How do you use x" issues are usability issues.  If 
you are an end-user of the product and have a comment or question then use 
u...@metron.incubator.apache.org.  If you have a problem and a strong suspicion 
that you might have found a bug, please cross-reference 
dev@metron.incubator.apache.org as well
I don't understand the UI, what does button x do?
What should the output of function x be?
It would be nice if I had feature x along with feature y
etc...

If you found a security-related issue, please report immediately to 
secur...@metron.incubator.apache.org. Please adhere to the following Apache 
policy found here. DO NOT FILE A JIRA, DO NOT POST ON ANY OTHER BOARD
I can get access to data I should not have access to
I have privileges to do things I should not be allowed to do
I found that this project is susceptible to an exploit 
etc...

Please report issues related to the JIRA/Wiki to 
iss...@metron.incubator.apache.org
I don't have access to create/assign JIRAs to myself
I don't have visibility/access to certain JIRA featuers
I can't create or view a wiki entry
etc

20.12.2016, 13:13, "Matt Foley" :
> Suggestions:
>
> For the first two categories (dev and user), the word “issue” is ambiguous. 
> We should distinguish between “I have a problem” and “I have found a bug”. “I 
> have a problem” might mean I need advice, or it might mean I’ve found a bug; 
> and in this case it’s appropriate to bring it to the mailing lists for 
> comment and advice first. But if it’s clear they’ve found a bug (or if it 
> becomes clear after interaction on the lists) then they should file a Jira, 
> with possible follow-up on the mailing lists if they want advice regarding a 
> local fix or work-around.
>
> Also, mention that messages should not be cross-posted between dev and user 
> lists. Go ahead and pick one, and community members will move the posting if 
> it is better suited to the other list.
>
> Thanks,
> --Matt
>
> On 12/20/16, 10:27 AM, "James Sirota"  wrote:
>
> As a criterion for exiting incubation we need clear documentation for how 
> to report Metron issues. What do you think about this?
>
> If you find issues with Metron please report them to the Metron community.
> Please report all developer-level issues to 
> dev@metron.incubator.apache.org. Examples of developer issues would be:
> Project fails to compile or is failing unit or integration tests
> Project or individual components fail to install in my environment
> There are error messages in my logs
> I need help with coding or extending a specific component
> etc...
>
> If you are an end-user of the product and have a comment or question then 
> use u...@metron.incubator.apache.org
> I don't understand the UI, what does button x do?
> What should the output of function x be?
> It would be nice if I had feature x along with feature y
> etc...
>
> If you found a security-related issue, please report immediately to 
> secur...@metron.incubator.apache.org. DO NOT FILE A JIRA, DO NOT POST ON ANY 
> OTHER BOARD
> I can get access to data I should not have access to
> I have privileges to do things I should not be allowed to do
> I found that this project is susceptible to an exploit
> etc...
>
> Please report issues related to the JIRA/Wiki to 
> iss...@metron.incubator.apache.org
> I don't have access to create/assign JIRAs to myself
> I don't have visibility/access to certain JIRA featuers
> I can't create or view a wiki entry
> etc
>
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

[GitHub] incubator-metron issue #408: METRON-608 Mpack to install a single-node test ...

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/408
  
Oops, now the 2017 fix (METRON-647) is committed.  Kicking Travis again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #408: METRON-608 Mpack to install a single-nod...

Github user mattf-horton closed the pull request at:

https://github.com/apache/incubator-metron/pull/408


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Podling Report Reminder - January 2017

Filed the following report:


Metron

Metron is a project dedicated to providing an extensible and scalable advanced
network security analytics tool. It has strong foundations in the Apache
Hadoop ecosystem.

Metron has been incubating since 2015-12-06.

Three most important issues to address in the move towards graduation:

  1. Trigger a community discussion to graduate from incubation.  We have 
received a lot of value from the incubation process and after speaking to our 
mentors and going through the Apache Project Maturity Model we believe that we 
will soon be ready to graduate.  
  2. Vote to ratify our Development Guidelines and Process for Reporting Issues 
  3. Vote to ratify our Release Process 

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?


We refreshed our website to be more compliant with Apache policies 

How has the community developed since the last report?

- We added 2 committers 
- We had a clean Apache build

How has the project developed since the last report?

- We closed 63 PRs as of our last build
- We integrated with Apache Ambari to make the project easier to deploy
- We modified and re-voted on our Bylaws 
- We successfully filed our Name Search Jira.  

Date of last release:

  2016-11-14

When were the last committers or PPMC members elected?

  2016-9-1

Signed-off-by:

  [ ](metron) Billie Rinaldi
  [ ](metron) Chris Mattmann
  [ ](metron) Owen O'Malley
  [ ](metron) P. Taylor Goetz
  [ ](metron) Vinod Kumar Vavilapalli

Shepherd/Mentor notes:

02.01.2017, 10:12, "johndam...@apache.org" :
> Dear podling,
>
> This email was sent by an automated system on behalf of the Apache
> Incubator PMC. It is an initial reminder to give you plenty of time to
> prepare your quarterly board report.
>
> The board meeting is scheduled for Wed, 18 January 2017, 10:30 am PDT.
> The report for your podling will form a part of the Incubator PMC
> report. The Incubator PMC requires your report to be submitted 2 weeks
> before the board meeting, to allow sufficient time for review and
> submission (Wed, January 04).
>
> Please submit your report with sufficient time to allow the Incubator
> PMC, and subsequently board members to review and digest. Again, the
> very latest you should submit your report is 2 weeks prior to the board
> meeting.
>
> Thanks,
>
> The Apache Incubator PMC
>
> Submitting your Report
>
> --
>
> Your report should contain the following:
>
> * Your project name
> * A brief description of your project, which assumes no knowledge of
> the project or necessarily of its field
> * A list of the three most important issues to address in the move
> towards graduation.
> * Any issues that the Incubator PMC or ASF Board might wish/need to be
> aware of
> * How has the community developed since the last report
> * How has the project developed since the last report.
>
> This should be appended to the Incubator Wiki page at:
>
> https://wiki.apache.org/incubator/January2017
>
> Note: This is manually populated. You may need to wait a little before
> this page is created from a template.
>
> Mentors
> ---
>
> Mentors should review reports for their project(s) and sign them off on
> the Incubator wiki page. Signing off reports shows that you are
> following the project - projects that are not signed may raise alarms
> for the Incubator PMC.
>
> Incubator PMC

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

[GitHub] incubator-metron issue #409: METRON-644 RPM builds only work with Docker for...

2017-01-04 Thread justinleet

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/409
  
Thanks for doing this, it'll be really nice to not be dependent on Docker 
for Mac for this.  Could you also update the prereqs (and any other necessary 
docs) in README.md in metron-deployment?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #400: METRON-636: Capture memory and cpu details as a...

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/400
  
Thanks for the explanation @anandsubbu .  Glad we agree on going after 
'static' system information.  That makes sense to me.

What information specifically do you want to capture?  Amount of physical 
memory? Number of CPU cores?

As is, the script spits out more dynamic information than static, in my 
opinion.  Maybe the script could filter out some of the other elements that 
might be more distraction than useful.  For example, on my Mac...
```
$ top -l 1 | head -10 2>/dev/null
Processes: 363 total, 2 running, 361 sleeping, 1735 threads
2017/01/04 12:42:04
Load Avg: 2.19, 1.87, 1.55
CPU usage: 22.54% user, 19.7% sys, 58.38% idle
SharedLibs: 167M resident, 37M data, 45M linkedit.
MemRegions: 129777 total, 5335M resident, 189M private, 1860M shared.
PhysMem: 13G used (2764M wired), 2753M unused.
VM: 1076G vsize, 620M framework vsize, 3733004(0) swapins, 3839813(0) 
swapouts.
Networks: packets: 7978036/7950M in, 4748808/532M out.
Disks: 7485876/92G read, 2696358/81G written.
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: METRON-648 GrokWebSphereParserTest and BasicAsaParserTest are not 2017-safe

2017-01-04 Thread Justin Leet

This should be taken care of now that https://github.com/
apache/incubator-metron/pull/412 is merged into Apache master.

The follow-on Jira for handling dates without years as seen in the
GrokWebSphereParser is at https://issues.apache.org/jira/browse/METRON-649.

Justin

On Wed, Jan 4, 2017 at 11:31 AM, Kyle Richardson 
wrote:

> +1 Why didn't I think of that? :) Thanks, Justin.
>
> -Kyle
>
> On Wed, Jan 4, 2017 at 10:26 AM, Nick Allen  wrote:
>
> > +1 We can't merge anything else until we address this.  Thanks for
> > volunteering, Justin.
> >
> > On Wed, Jan 4, 2017 at 10:24 AM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > +1
> > >
> > > On Wed, Jan 4, 2017 at 8:04 AM, Justin Leet 
> > wrote:
> > >
> > > > In the short term, we could just generate the timestamp appropriately
> > > with
> > > > the current year in the test for the test and spin off another JIRA
> for
> > > > actually addressing the question of what we do with this data (Keep
> in
> > > mind
> > > > we can eventually have replay use cases, so assuming the past year
> > might
> > > > not be totally sufficient either.)
> > > >
> > > > At that point it'll at least be year agnostic, but probably not the
> > > actual
> > > > output we want. Normally, I'd rather it be handled correctly, but
> given
> > > > that our builds fail, I'd rather have something less broken until we
> > get
> > > a
> > > > more correct solution.
> > > >
> > > > I can take care of doing that today.  Any objections to that
> solution?
> > > >
> > > > Justin
> > > >
> > > > On Wed, Jan 4, 2017 at 9:34 AM, Kyle Richardson <
> > > kylerichards...@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Unfortunately, it's not going to be quite as simple as just adding
> > the
> > > > year
> > > > > into the test strings, at least for GrokWebSphereParserTest. (For
> > > > > BasicAsaParserTest, updating the test string worked just fine.)
> > > > >
> > > > > It turns out that that grok pattern being used only expects the
> month
> > > and
> > > > > day in the timestamp of the syslog messages. I'm happy to a take a
> > stab
> > > > and
> > > > > making it year safe by reusing some of the code from the
> > > BasicAsaParser;
> > > > > however, I have limited time today and it will likely take me until
> > > > Friday
> > > > > to get a PR submitted given the new scope of changes required.
> > > > >
> > > > > -Kyle
> > > > >
> > > > > On Wed, Jan 4, 2017 at 12:50 AM, Matt Foley 
> > wrote:
> > > > >
> > > > > > Yes, this is an endemic problem with log processing.  And I agree
> > > > adding
> > > > > > the year to the testString is the best fix for our short-term
> > > problem.
> > > > > >
> > > > > > For future consideration, we should consider if there should be
> an
> > > > > > assumption/preference in the parser that the logs are in the
> > “past”.
> > > > > > Granted, if the timezone is also unspecified, there is still a 24
> > hr
> > > > > period
> > > > > > of uncertainty, but it does seem that on Jan 3 2017 the preferred
> > > > > > interpretation of “Apr 15” would be Apr 15 2016, not 2017.
> > > > > >
> > > > > > Cheers,
> > > > > > --Matt
> > > > > >
> > > > > > On 1/3/17, 5:14 PM, "Michael Miklavcic" <
> > michael.miklav...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > I also introduced a Clock object and testing mechanism back
> in
> > > > > > METRON-235 -
> > > > > > https://github.com/apache/incubator-metron/pull/156
> > > > > > Sample test utilizing the Clock object here -
> > > > > > https://github.com/apache/incubator-metron/blob/master/
> > > > > > metron-platform/metron-pcap-backend/src/test/java/org/
> > > > > > apache/metron/pcap/query/PcapCliTest.java
> > > > > >
> > > > > > That being said, it's probably better to use the new
> java.time
> > > > fixed
> > > > > > clock
> > > > > > implementation in all places, as referenced by Matt. I'm
> agreed
> > > > with
> > > > > > everyone on a quick fix for the build and a follow-on PR to
> > > > introduce
> > > > > > appropriate dep injection for testing.
> > > > > >
> > > > > > AFA string dates with no year, we had something similar show
> up
> > > in
> > > > > the
> > > > > > Snort parser. There ended up being a configuration option in
> > > Snort
> > > > to
> > > > > > enable a year to be printed, but we may want to offer
> > > alternatives
> > > > > for
> > > > > > other parsers. Regardless of how we approach this it gets
> messy
> > > > when
> > > > > > you
> > > > > > start thinking about potentially different src/dest timezones
> > > > across
> > > > > a
> > > > > > new
> > > > > > year boundary in addition to data replay. I would urge our
> main
> > > > goal
> > > > > > here
> > > > > > to be idempotency.
> > > > > >
> > > > > > Best,
> > > > > > Mike
> > > > > >
> > > > > > On Tue, Jan 3, 2017 at 5:05 PM, Kyle

[GitHub] incubator-metron issue #400: METRON-636: Capture memory and cpu details as a...

2017-01-04 Thread anandsubbu

Github user anandsubbu commented on the issue:

https://github.com/apache/incubator-metron/pull/400
  
Hi @nickwallen , thank you for the review. 

At the outset, getting the system CPU and Memory info would help 
troubleshooting, especially for single node deployments which are attempted on 
environments with scarce resources (This 
[thread](https://community.hortonworks.com/questions/51666/apache-metron-single-node-installation-problems.html)
 for example).

My intention is in fact to capture the 'static' system information. I agree 
with the challenges associated with capturing the runtime information--which 
this script is not intended to do.

My initial approach was to dump the contents of `/proc/cpuinfo` and 
`/proc/meminfo`. However, for lack of the same equivalent on Mac, I resorted to 
getting the first few lines of `top` (again with different switches for Linux 
and Mac :/). 

I can still dump the output of /proc/cpuinfo and /proc/meminfo on Linux and 
use a different alternative for Mac. I looked as much but could not find one 
for Mac. Please let me know if you are aware of one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #411: METRON-635: Vagrant provisioning fails on CentO...

2017-01-04 Thread JonZeolla

Github user JonZeolla commented on the issue:

https://github.com/apache/incubator-metron/pull/411

@nickwallen Right. So more granular details about how I tested:
1. Installed [CentOS 6.8
minimal](http://centos.pymesolutionsweb.com/7/isos/x86_64/CentOS-7-x86_64-Minimal-1611.iso)
1. `yum install -f git`
1. `git clone https://github.com/jonzeolla/development`
1. `cd development/Bash && chmod 755 setupMetron.sh`
1. `./setupMetron.sh -fvs full`
* This should not show an issue.
1. Destroy the vagrant instance (see
`/usr/local/metron/*/metron-deployment/vagrant/full-dev-platform`)
1. `./setupMetron.sh -fvst full`
* You should get the issue.
* Adding `-t` prevents `setupMetron.sh` from adding `scp_if_ssh` to
`ansible.cfg`, and you will see the error. For the most pristine testing, I
reinstalled the OS fresh for each run of `setupMetron.sh` (hence why testing
this was a bit of a pain). Finally:
1. Kill the vagrant instance again
1. `./setupMetron.sh -fvst -p 411 full`
* This runs the setup process without adding scp_if_ssh, but it merges
in this PR before building Metron and starting vagrant. You should not see the
issue.

Notes:
* `full` can be substituted with `quick`, `fastcapa`, or `codelab` for
testing the different vagrant deploy methods. As you're aware, for a good
fastcapa build, this requires `-p 410`.
* Obviously it's probably worth reviewing `setupMetron.sh` to see exactly
what's happening, but the `-v` provides some amount of feedback.
* There is a `-d` flag which adds `ansible.verbose = ""` to the
Vagrantfile prior to doing `vagrant up` and then removes it after completion
for additional troubleshooting.
* All the script really does is setup up all the dependencies. From time
to time, running `setupMetron.sh` will fail because something upstream is being
sync'd or performing maintenance/having issues, and a rerun usually fixes the
issue.
* The script is currently super inefficient and doesn't check to see if
something's already installed before attempting to install it.

Happy to help provide any additional detail as would be helpful. I wasn't
able to pin this down exactly, but I have seen 3 others independently report
this issue and the fix worked for them.

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #412: METRON-647 Parser unit test failures due...

2017-01-04 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/412


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #400: METRON-636: Capture memory and cpu details as a...

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/400
  
Hi @anandsubbu - Under what scenarios do you think this additional 
information would help?

The script was primarily made to capture 'static' bits of platform 
information that might prevent a successful build or deployment of Metron.

The changes you have made have added some 'dynamic' bits; aka CPU, memory.  
For these changes to help, the script would have to be run exactly when there 
are high CPU/memory conditions.  I don't know if this is possible.  

What I have seen happen is that the guest OS maxes out memory and starts 
harvesting/killing processes.  This effectively kills some service needed by 
Metron and prevents Metron from working.  But once the processes are killed, 
the memory use drops.  So if I am having problems with Metron and run the 
script it is just as likely to report no memory problems, even though it was 
actually the lack of memory that caused the problem.

Maybe I am thinking about the use case wrong here.  Please correct me if 
you disagree.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #408: METRON-608 Mpack to install a single-nod...

Github user mattf-horton closed the pull request at:

https://github.com/apache/incubator-metron/pull/408


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #408: METRON-608 Mpack to install a single-nod...

GitHub user mattf-horton reopened a pull request:

https://github.com/apache/incubator-metron/pull/408

METRON-608 Mpack to install a single-node test cluster

This "metron-mpack-singlenode" is very similar to "metron-mpack", and can 
be usefully compared with it for code review, altho it also includes many bug 
fixes proposed in METRON-634.  As it says in the Jira, this is a short-term fix 
by providing a completely separate Mpack just for the single-node scenario, 
until we can work the bugs out of an mpack that works on all sizes of cluster.

This was tested on a Centos7 VM with 16GB of RAM, and all prerequisites 
installed, including Python 2.7.11.  The usual install process was followed, 
with Ambari 2.4.2 and HDP stack 2.5.3.  After successful startup, a few 
thousand 'bro' packets were injected, and were seen to correctly propagate 
through parser, enricher, and indexer topologies, and finally into 
elasticsearch.

An outstanding issue is that the configuration validator in the mpack 
doesn't work.  It fails with "unknown error" and must be ignored.  The 
multi-node mpack already had this problem, and I have run out of time to try to 
fix it here.  Consequently, my enhancement of trying to automatically flag the 
user if less than five Storm Supervisor Slots are allocated (commit 9ad8706), 
is untested.  I would like to open a new Jira for this issue, since it was 
pre-existing before these additions.
Thanks,
--Matt

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattf-horton/incubator-metron METRON-608

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/408.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #408


commit 05571b501222b91457bd2f1b2ccc096af015f4cb
Author: mattf-horton 
Date:   2016-12-08T05:31:40Z

METRON-608 Mpack to install a single-node test cluster

commit f2245900cd94389bc1c004c60f1606a49ca6ea8c
Author: mattf-horton 
Date:   2016-12-10T01:01:41Z

Add quicklinks to Elasticsearch Service page in Ambari, for health and 
indexes.

commit 542a6f01376a656b539040992e9fa727efd8a047
Author: mattf-horton 
Date:   2016-12-10T01:02:40Z

METRON-608 fix several bugs, especially in METRON use of 'es_url'

commit fb646fdda345107b0debd8395743630cc5e0
Author: mattf-horton 
Date:   2016-12-10T09:05:43Z

update version to 0.3.0

commit 8e30344e0b50fdff0cf8a1a66e7f41868a384171
Author: mattf-horton 
Date:   2016-12-10T09:09:30Z

Merge/rebase updates from 0.3.0

commit 9ad8706e5209ca9506710ca6498ac387e667ddf0
Author: mattf-horton 
Date:   2016-12-13T02:03:07Z

METRON-608 add service_advisor validation for number of Storm slots, 
enhance README, and a couple bug fixes

commit d4c97e6de4838265b72139727a5fd1082388117b
Author: mattf-horton 
Date:   2016-12-13T05:48:22Z

tweak pom.xml file

commit 95d83702ef844bc94c2ce54bccaebb7b7cb0513a
Author: mattf-horton 
Date:   2016-12-20T00:54:07Z

METRON-608 multiple small bug fixes

commit b5b3a348527c03bc778a5fc5dd0d94ec3a00470b
Author: mattf-horton 
Date:   2016-12-20T01:39:37Z

rebase to Dec 19, 2016




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #408: METRON-608 Mpack to install a single-node test ...

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/408
  
kicking travis


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #412: METRON-647 Parser unit test failures due to ass...

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/412
  
+1 Looks good to me


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #411: METRON-635: Vagrant provisioning fails on CentO...

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/411
  
@JonZeolla I want to make sure I understand under what conditions this bug 
occurs.  If I try to provision Quick Dev when running on a CentOS 6.8 host, I 
will hit this error?

Any idea why this happens?  Is it a difference between the Ansible build on 
Mac versus CentOS?  Or something else entirely?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: METRON-648 GrokWebSphereParserTest and BasicAsaParserTest are not 2017-safe

2017-01-04 Thread Kyle Richardson

+1 Why didn't I think of that? :) Thanks, Justin.

-Kyle

On Wed, Jan 4, 2017 at 10:26 AM, Nick Allen  wrote:

> +1 We can't merge anything else until we address this.  Thanks for
> volunteering, Justin.
>
> On Wed, Jan 4, 2017 at 10:24 AM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > +1
> >
> > On Wed, Jan 4, 2017 at 8:04 AM, Justin Leet 
> wrote:
> >
> > > In the short term, we could just generate the timestamp appropriately
> > with
> > > the current year in the test for the test and spin off another JIRA for
> > > actually addressing the question of what we do with this data (Keep in
> > mind
> > > we can eventually have replay use cases, so assuming the past year
> might
> > > not be totally sufficient either.)
> > >
> > > At that point it'll at least be year agnostic, but probably not the
> > actual
> > > output we want. Normally, I'd rather it be handled correctly, but given
> > > that our builds fail, I'd rather have something less broken until we
> get
> > a
> > > more correct solution.
> > >
> > > I can take care of doing that today.  Any objections to that solution?
> > >
> > > Justin
> > >
> > > On Wed, Jan 4, 2017 at 9:34 AM, Kyle Richardson <
> > kylerichards...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Unfortunately, it's not going to be quite as simple as just adding
> the
> > > year
> > > > into the test strings, at least for GrokWebSphereParserTest. (For
> > > > BasicAsaParserTest, updating the test string worked just fine.)
> > > >
> > > > It turns out that that grok pattern being used only expects the month
> > and
> > > > day in the timestamp of the syslog messages. I'm happy to a take a
> stab
> > > and
> > > > making it year safe by reusing some of the code from the
> > BasicAsaParser;
> > > > however, I have limited time today and it will likely take me until
> > > Friday
> > > > to get a PR submitted given the new scope of changes required.
> > > >
> > > > -Kyle
> > > >
> > > > On Wed, Jan 4, 2017 at 12:50 AM, Matt Foley 
> wrote:
> > > >
> > > > > Yes, this is an endemic problem with log processing.  And I agree
> > > adding
> > > > > the year to the testString is the best fix for our short-term
> > problem.
> > > > >
> > > > > For future consideration, we should consider if there should be an
> > > > > assumption/preference in the parser that the logs are in the
> “past”.
> > > > > Granted, if the timezone is also unspecified, there is still a 24
> hr
> > > > period
> > > > > of uncertainty, but it does seem that on Jan 3 2017 the preferred
> > > > > interpretation of “Apr 15” would be Apr 15 2016, not 2017.
> > > > >
> > > > > Cheers,
> > > > > --Matt
> > > > >
> > > > > On 1/3/17, 5:14 PM, "Michael Miklavcic" <
> michael.miklav...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > I also introduced a Clock object and testing mechanism back in
> > > > > METRON-235 -
> > > > > https://github.com/apache/incubator-metron/pull/156
> > > > > Sample test utilizing the Clock object here -
> > > > > https://github.com/apache/incubator-metron/blob/master/
> > > > > metron-platform/metron-pcap-backend/src/test/java/org/
> > > > > apache/metron/pcap/query/PcapCliTest.java
> > > > >
> > > > > That being said, it's probably better to use the new java.time
> > > fixed
> > > > > clock
> > > > > implementation in all places, as referenced by Matt. I'm agreed
> > > with
> > > > > everyone on a quick fix for the build and a follow-on PR to
> > > introduce
> > > > > appropriate dep injection for testing.
> > > > >
> > > > > AFA string dates with no year, we had something similar show up
> > in
> > > > the
> > > > > Snort parser. There ended up being a configuration option in
> > Snort
> > > to
> > > > > enable a year to be printed, but we may want to offer
> > alternatives
> > > > for
> > > > > other parsers. Regardless of how we approach this it gets messy
> > > when
> > > > > you
> > > > > start thinking about potentially different src/dest timezones
> > > across
> > > > a
> > > > > new
> > > > > year boundary in addition to data replay. I would urge our main
> > > goal
> > > > > here
> > > > > to be idempotency.
> > > > >
> > > > > Best,
> > > > > Mike
> > > > >
> > > > > On Tue, Jan 3, 2017 at 5:05 PM, Kyle Richardson <
> > > > > kylerichards...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Agreed. I prefer the quick win to get us back to successful
> > > builds.
> > > > > >
> > > > > > I do think it's worth a general discussion around how we want
> > to
> > > > > handle
> > > > > > the parsing of string dates with no year. In the long run,
> > Matt's
> > > > > > suggestion of incorporating the Clock object is probably the
> > > route
> > > > > to go;
> > > > > > albeit as a separate enhancement PR.
> > > > > >
> > > > > > I'll start a new discuss thread for that and submit a PR for
> > the
> > > >

[GitHub] incubator-metron pull request #412: METRON-647 Parser unit test failures due...

2017-01-04 Thread justinleet

GitHub user justinleet opened a pull request:

https://github.com/apache/incubator-metron/pull/412

METRON-647 Parser unit test failures due to assumed year values

Short term fix for the build, along with some cleanup noted in METRON-648 
(which was closed as a dupe of 647).

Two classes are adjusted:

`BasicAsaParserTest:testIp6Addr`: The rawMessage string has been adjusted 
to include 2016 explicitly.  In addition `assertTrue` has been replaced with 
`assertEquals` and params put in correct order.

`GrokWebSphereParserTest`: Tests no longer use explicit timestamps, but 
instead generate a timestamp based on the current year (UTC timezone is used, 
because GrokParser defaults to UTC if no timezone is explicitly provided). 
Using current year is not desirable long term behavior, but the tests now match 
existing behavior and the build can succeed. In addition the order of params to 
`assertEquals` is corrected and values are no longer unnecessarily coerced.

A follow-on ticket will be created for handling dates in a more robust 
manner.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/justinleet/incubator-metron METRON-647

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/412.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #412


commit 713e0297ff324d2a0a478ca89d600b05beb609c1
Author: justinjleet 
Date:   2017-01-04T15:49:10Z

Altering unit tests to handle it being not 2016




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: METRON-648 GrokWebSphereParserTest and BasicAsaParserTest are not 2017-safe

2017-01-04 Thread Nick Allen

+1 We can't merge anything else until we address this.  Thanks for
volunteering, Justin.

On Wed, Jan 4, 2017 at 10:24 AM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> +1
>
> On Wed, Jan 4, 2017 at 8:04 AM, Justin Leet  wrote:
>
> > In the short term, we could just generate the timestamp appropriately
> with
> > the current year in the test for the test and spin off another JIRA for
> > actually addressing the question of what we do with this data (Keep in
> mind
> > we can eventually have replay use cases, so assuming the past year might
> > not be totally sufficient either.)
> >
> > At that point it'll at least be year agnostic, but probably not the
> actual
> > output we want. Normally, I'd rather it be handled correctly, but given
> > that our builds fail, I'd rather have something less broken until we get
> a
> > more correct solution.
> >
> > I can take care of doing that today.  Any objections to that solution?
> >
> > Justin
> >
> > On Wed, Jan 4, 2017 at 9:34 AM, Kyle Richardson <
> kylerichards...@gmail.com
> > >
> > wrote:
> >
> > > Unfortunately, it's not going to be quite as simple as just adding the
> > year
> > > into the test strings, at least for GrokWebSphereParserTest. (For
> > > BasicAsaParserTest, updating the test string worked just fine.)
> > >
> > > It turns out that that grok pattern being used only expects the month
> and
> > > day in the timestamp of the syslog messages. I'm happy to a take a stab
> > and
> > > making it year safe by reusing some of the code from the
> BasicAsaParser;
> > > however, I have limited time today and it will likely take me until
> > Friday
> > > to get a PR submitted given the new scope of changes required.
> > >
> > > -Kyle
> > >
> > > On Wed, Jan 4, 2017 at 12:50 AM, Matt Foley  wrote:
> > >
> > > > Yes, this is an endemic problem with log processing.  And I agree
> > adding
> > > > the year to the testString is the best fix for our short-term
> problem.
> > > >
> > > > For future consideration, we should consider if there should be an
> > > > assumption/preference in the parser that the logs are in the “past”.
> > > > Granted, if the timezone is also unspecified, there is still a 24 hr
> > > period
> > > > of uncertainty, but it does seem that on Jan 3 2017 the preferred
> > > > interpretation of “Apr 15” would be Apr 15 2016, not 2017.
> > > >
> > > > Cheers,
> > > > --Matt
> > > >
> > > > On 1/3/17, 5:14 PM, "Michael Miklavcic"  >
> > > > wrote:
> > > >
> > > > I also introduced a Clock object and testing mechanism back in
> > > > METRON-235 -
> > > > https://github.com/apache/incubator-metron/pull/156
> > > > Sample test utilizing the Clock object here -
> > > > https://github.com/apache/incubator-metron/blob/master/
> > > > metron-platform/metron-pcap-backend/src/test/java/org/
> > > > apache/metron/pcap/query/PcapCliTest.java
> > > >
> > > > That being said, it's probably better to use the new java.time
> > fixed
> > > > clock
> > > > implementation in all places, as referenced by Matt. I'm agreed
> > with
> > > > everyone on a quick fix for the build and a follow-on PR to
> > introduce
> > > > appropriate dep injection for testing.
> > > >
> > > > AFA string dates with no year, we had something similar show up
> in
> > > the
> > > > Snort parser. There ended up being a configuration option in
> Snort
> > to
> > > > enable a year to be printed, but we may want to offer
> alternatives
> > > for
> > > > other parsers. Regardless of how we approach this it gets messy
> > when
> > > > you
> > > > start thinking about potentially different src/dest timezones
> > across
> > > a
> > > > new
> > > > year boundary in addition to data replay. I would urge our main
> > goal
> > > > here
> > > > to be idempotency.
> > > >
> > > > Best,
> > > > Mike
> > > >
> > > > On Tue, Jan 3, 2017 at 5:05 PM, Kyle Richardson <
> > > > kylerichards...@gmail.com>
> > > > wrote:
> > > >
> > > > > Agreed. I prefer the quick win to get us back to successful
> > builds.
> > > > >
> > > > > I do think it's worth a general discussion around how we want
> to
> > > > handle
> > > > > the parsing of string dates with no year. In the long run,
> Matt's
> > > > > suggestion of incorporating the Clock object is probably the
> > route
> > > > to go;
> > > > > albeit as a separate enhancement PR.
> > > > >
> > > > > I'll start a new discuss thread for that and submit a PR for
> the
> > > > quick fix.
> > > > >
> > > > > -Kyle
> > > > >
> > > > > > On Jan 3, 2017, at 5:20 PM, David Lyle  >
> > > > wrote:
> > > > > >
> > > > > > I'm not sure I'm an owner, but I have an opinion. :)
> > > > > >
> > > > > > I'd just add "2016". Easy and targeted.
> > > > > >
> > > > > > -D...
> > > > > >
> > > > > >
> > > > > >> On Tue, Jan

Re: METRON-648 GrokWebSphereParserTest and BasicAsaParserTest are not 2017-safe

2017-01-04 Thread Michael Miklavcic

+1

On Wed, Jan 4, 2017 at 8:04 AM, Justin Leet  wrote:

> In the short term, we could just generate the timestamp appropriately with
> the current year in the test for the test and spin off another JIRA for
> actually addressing the question of what we do with this data (Keep in mind
> we can eventually have replay use cases, so assuming the past year might
> not be totally sufficient either.)
>
> At that point it'll at least be year agnostic, but probably not the actual
> output we want. Normally, I'd rather it be handled correctly, but given
> that our builds fail, I'd rather have something less broken until we get a
> more correct solution.
>
> I can take care of doing that today.  Any objections to that solution?
>
> Justin
>
> On Wed, Jan 4, 2017 at 9:34 AM, Kyle Richardson  >
> wrote:
>
> > Unfortunately, it's not going to be quite as simple as just adding the
> year
> > into the test strings, at least for GrokWebSphereParserTest. (For
> > BasicAsaParserTest, updating the test string worked just fine.)
> >
> > It turns out that that grok pattern being used only expects the month and
> > day in the timestamp of the syslog messages. I'm happy to a take a stab
> and
> > making it year safe by reusing some of the code from the BasicAsaParser;
> > however, I have limited time today and it will likely take me until
> Friday
> > to get a PR submitted given the new scope of changes required.
> >
> > -Kyle
> >
> > On Wed, Jan 4, 2017 at 12:50 AM, Matt Foley  wrote:
> >
> > > Yes, this is an endemic problem with log processing.  And I agree
> adding
> > > the year to the testString is the best fix for our short-term problem.
> > >
> > > For future consideration, we should consider if there should be an
> > > assumption/preference in the parser that the logs are in the “past”.
> > > Granted, if the timezone is also unspecified, there is still a 24 hr
> > period
> > > of uncertainty, but it does seem that on Jan 3 2017 the preferred
> > > interpretation of “Apr 15” would be Apr 15 2016, not 2017.
> > >
> > > Cheers,
> > > --Matt
> > >
> > > On 1/3/17, 5:14 PM, "Michael Miklavcic" 
> > > wrote:
> > >
> > > I also introduced a Clock object and testing mechanism back in
> > > METRON-235 -
> > > https://github.com/apache/incubator-metron/pull/156
> > > Sample test utilizing the Clock object here -
> > > https://github.com/apache/incubator-metron/blob/master/
> > > metron-platform/metron-pcap-backend/src/test/java/org/
> > > apache/metron/pcap/query/PcapCliTest.java
> > >
> > > That being said, it's probably better to use the new java.time
> fixed
> > > clock
> > > implementation in all places, as referenced by Matt. I'm agreed
> with
> > > everyone on a quick fix for the build and a follow-on PR to
> introduce
> > > appropriate dep injection for testing.
> > >
> > > AFA string dates with no year, we had something similar show up in
> > the
> > > Snort parser. There ended up being a configuration option in Snort
> to
> > > enable a year to be printed, but we may want to offer alternatives
> > for
> > > other parsers. Regardless of how we approach this it gets messy
> when
> > > you
> > > start thinking about potentially different src/dest timezones
> across
> > a
> > > new
> > > year boundary in addition to data replay. I would urge our main
> goal
> > > here
> > > to be idempotency.
> > >
> > > Best,
> > > Mike
> > >
> > > On Tue, Jan 3, 2017 at 5:05 PM, Kyle Richardson <
> > > kylerichards...@gmail.com>
> > > wrote:
> > >
> > > > Agreed. I prefer the quick win to get us back to successful
> builds.
> > > >
> > > > I do think it's worth a general discussion around how we want to
> > > handle
> > > > the parsing of string dates with no year. In the long run, Matt's
> > > > suggestion of incorporating the Clock object is probably the
> route
> > > to go;
> > > > albeit as a separate enhancement PR.
> > > >
> > > > I'll start a new discuss thread for that and submit a PR for the
> > > quick fix.
> > > >
> > > > -Kyle
> > > >
> > > > > On Jan 3, 2017, at 5:20 PM, David Lyle 
> > > wrote:
> > > > >
> > > > > I'm not sure I'm an owner, but I have an opinion. :)
> > > > >
> > > > > I'd just add "2016". Easy and targeted.
> > > > >
> > > > > -D...
> > > > >
> > > > >
> > > > >> On Tue, Jan 3, 2017 at 5:08 PM, Matt Foley 
> > > wrote:
> > > > >>
> > > > >> I’ll subordinate this to METRON-647 since it was evidently
> filed
> > > while I
> > > > >> was writing METRON-648 (I did check before!)
> > > > >>
> > > > >> The question below remains valid, however…
> > > > >>
> > > > >>
> > > > >> On 1/3/17, 1:59 PM, "Matt Foley"  wrote:
> > > > >>
> > > > >>Hi all,
> > > > >>

Re: METRON-648 GrokWebSphereParserTest and BasicAsaParserTest are not 2017-safe

2017-01-04 Thread Justin Leet

In the short term, we could just generate the timestamp appropriately with
the current year in the test for the test and spin off another JIRA for
actually addressing the question of what we do with this data (Keep in mind
we can eventually have replay use cases, so assuming the past year might
not be totally sufficient either.)

At that point it'll at least be year agnostic, but probably not the actual
output we want. Normally, I'd rather it be handled correctly, but given
that our builds fail, I'd rather have something less broken until we get a
more correct solution.

I can take care of doing that today.  Any objections to that solution?

Justin

On Wed, Jan 4, 2017 at 9:34 AM, Kyle Richardson 
wrote:

> Unfortunately, it's not going to be quite as simple as just adding the year
> into the test strings, at least for GrokWebSphereParserTest. (For
> BasicAsaParserTest, updating the test string worked just fine.)
>
> It turns out that that grok pattern being used only expects the month and
> day in the timestamp of the syslog messages. I'm happy to a take a stab and
> making it year safe by reusing some of the code from the BasicAsaParser;
> however, I have limited time today and it will likely take me until Friday
> to get a PR submitted given the new scope of changes required.
>
> -Kyle
>
> On Wed, Jan 4, 2017 at 12:50 AM, Matt Foley  wrote:
>
> > Yes, this is an endemic problem with log processing.  And I agree adding
> > the year to the testString is the best fix for our short-term problem.
> >
> > For future consideration, we should consider if there should be an
> > assumption/preference in the parser that the logs are in the “past”.
> > Granted, if the timezone is also unspecified, there is still a 24 hr
> period
> > of uncertainty, but it does seem that on Jan 3 2017 the preferred
> > interpretation of “Apr 15” would be Apr 15 2016, not 2017.
> >
> > Cheers,
> > --Matt
> >
> > On 1/3/17, 5:14 PM, "Michael Miklavcic" 
> > wrote:
> >
> > I also introduced a Clock object and testing mechanism back in
> > METRON-235 -
> > https://github.com/apache/incubator-metron/pull/156
> > Sample test utilizing the Clock object here -
> > https://github.com/apache/incubator-metron/blob/master/
> > metron-platform/metron-pcap-backend/src/test/java/org/
> > apache/metron/pcap/query/PcapCliTest.java
> >
> > That being said, it's probably better to use the new java.time fixed
> > clock
> > implementation in all places, as referenced by Matt. I'm agreed with
> > everyone on a quick fix for the build and a follow-on PR to introduce
> > appropriate dep injection for testing.
> >
> > AFA string dates with no year, we had something similar show up in
> the
> > Snort parser. There ended up being a configuration option in Snort to
> > enable a year to be printed, but we may want to offer alternatives
> for
> > other parsers. Regardless of how we approach this it gets messy when
> > you
> > start thinking about potentially different src/dest timezones across
> a
> > new
> > year boundary in addition to data replay. I would urge our main goal
> > here
> > to be idempotency.
> >
> > Best,
> > Mike
> >
> > On Tue, Jan 3, 2017 at 5:05 PM, Kyle Richardson <
> > kylerichards...@gmail.com>
> > wrote:
> >
> > > Agreed. I prefer the quick win to get us back to successful builds.
> > >
> > > I do think it's worth a general discussion around how we want to
> > handle
> > > the parsing of string dates with no year. In the long run, Matt's
> > > suggestion of incorporating the Clock object is probably the route
> > to go;
> > > albeit as a separate enhancement PR.
> > >
> > > I'll start a new discuss thread for that and submit a PR for the
> > quick fix.
> > >
> > > -Kyle
> > >
> > > > On Jan 3, 2017, at 5:20 PM, David Lyle 
> > wrote:
> > > >
> > > > I'm not sure I'm an owner, but I have an opinion. :)
> > > >
> > > > I'd just add "2016". Easy and targeted.
> > > >
> > > > -D...
> > > >
> > > >
> > > >> On Tue, Jan 3, 2017 at 5:08 PM, Matt Foley 
> > wrote:
> > > >>
> > > >> I’ll subordinate this to METRON-647 since it was evidently filed
> > while I
> > > >> was writing METRON-648 (I did check before!)
> > > >>
> > > >> The question below remains valid, however…
> > > >>
> > > >>
> > > >> On 1/3/17, 1:59 PM, "Matt Foley"  wrote:
> > > >>
> > > >>Hi all,
> > > >>As described in https://issues.apache.org/
> > jira/browse/METRON-648 ,
> > > >> these two test modules are not year-safe, and are suddenly (as
> of
> > 2017)
> > > >> giving false Travis errors.
> > > >>
> > > >>I can fix it quickly, but a question for the “owners” of
> > GrokParser:
> > > >> Do you have an opinion as

[GitHub] incubator-metron issue #407: METRON-643: Stellar function documentation need...