Re: [MENTORS] Release Maturity

[larry mccay]

IMO, such a meeting is not necessary.
Care must also be taken to make no decisions in these meetings where the
entire community doesn't have a say - even if they are allowed to attend.
Timezones and other priorities make such meetings difficult for many.

Options can be discussed and presented to the dev@ list as a summary of the
off-list discussion and the community still makes the decisions.

What I like is to see a planning email for the next release that calls out
specific driving features as a theme for the next release the corresponding
JIRAs have the Fix Version set to that planned release.

As bugs are discovered and fixed they are added to the JIRAs with the
release Fix Version along with the driving features.

As close down seems eminent, a follow up email is sent to inform of the
close down period and folks can step up for JIRAs that are still
outstanding, ask to delay for some additional time, etc. I find this to be
useful at a week or so out from the first RC.

Just my two cents.


On Wed, Oct 19, 2016 at 8:08 PM, Kyle Richardson 
wrote:

> I'm +1 on a meeting to discuss the backlog and would suggest, to be
> considered, each JIRA should have a clear description. I think that 72
> hours is good for final adds to an upcoming release.
>
> I personally like the idea of having more visibility on which JIRAs are "up
> next" to help me figure out where contributions would be most valuable.
>
> -Kyle
>
> On Mon, Oct 17, 2016 at 1:50 PM, zeo...@gmail.com 
> wrote:
>
> > That's more aggressive than I would have initially suggested, but I would
> > be on board with that sort of a meeting.  Interested to see how others
> > feel.
> >
> > Jon
> >
> > On Mon, Oct 17, 2016 at 1:40 PM James Sirota  wrote:
> >
> > > Fair criticism.  Would you like to call a recurring meeting where PPMC
> > and
> > > community can get together and go through the Jira backlog?  We can
> then
> > > have the opportunity to triage the Jiras.  Do you think once a month
> > should
> > > be sufficient + 72 hours prior to the release to verify all desired
> Jiras
> > > are in?
> > >
> > > 17.10.2016, 10:01, "zeo...@gmail.com" :
> > > > I think that grooming the JIRA backlog at each release provides a
> good
> > > > method for new users or users less integrated into Metron development
> > to
> > > > understand the roadmap of the project by perusing the backlog. I feel
> > > > somewhat aware of the state of the Metron project but often have
> > > questions
> > > > about how prioritized certain issues are for development. I think the
> > > > easiest way to illustrate this gap are in these
> > > > <
> > > https://issues.apache.org/jira/browse/METRON-170?jql=
> > project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
> > 20fixVersion%20%3D%200.2.1BETA%20ORDER%20BY%20priority%20DESC
> > > >
> > > > two
> > > > <
> > > https://issues.apache.org/jira/browse/METRON-469?jql=
> > project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
> > 20fixVersion%20is%20EMPTY%20ORDER%20BY%20priority%20DESC
> > > >
> > > > links, where Metron currently has 45 unresolved issues slated for
> > > 0.2.1BETA
> > > > (?!?) and 71 unresolved issues that are unscheduled.
> > > >
> > > > I'd also like to see a comprehensive update of documentation on the
> > wiki
> > > >  (or
> > > > wherever it lives). Heck, TP2 isn't even on the releases page
> > > > , let
> > alone
> > > > 0.2.1 and other related details/changes.
> > > >
> > > > I'd be more than happy to help with either of those efforts, as
> > > applicable.
> > > >
> > > > Jon
> > > >
> > > > On Mon, Oct 17, 2016 at 11:39 AM Casey Stella 
> > > wrote:
> > > >
> > > >>  Hi Everyone,
> > > >>
> > > >>  I'd like to get a bit more systematic about how we release and I
> > wanted
> > > >>  some clarification and advice about suggested release process.
> > > >>
> > > >>  The last release, we
> > > >>
> > > >> - opened up the release via an announce thread that gave people
> > the
> > > >> opportunity to object and add JIRAs they felt were important to
> be
> > > >> considered for the release
> > > >> - made a release branch in git
> > > >> - made a release candidate tag
> > > >> - sent out the release candidate for a vote
> > > >> - when passed, sent the release candidate for a vote in general
> > > >>
> > > >>  A couple of questions:
> > > >>
> > > >> - Is 72 hours sufficient for people to suggest JIRAs that need
> to
> > > get in
> > > >> for the release?
> > > >> - What we did not do is have the JIRA backlog groomed and have
> > JIRAs
> > > >> assigned to releases beyond the current release. This would make
> > it
> > > >>  easier
> > > >> for people to find JIRAs that they want in. Is that a sensible
> > > >> 

Re: [DISCUSS] Improving quick-dev

[Jimson K James]

Really looking forward to dockerized Metron.
Thanks ahead.

On Wednesday, 19 October 2016, Kyle Richardson 
wrote:

> Sorry, I'm a bit late to the party on this one :).
>
> +1 on the four builds Nick described. Each would be useful and purpose
> built.
>
> I like the idea of using Docker, especially for local development and quick
> testing. Has anyone explored this? I'm envisioning very specific containers
> so you could spin up only the components you're actively working on.
> Certainly something I would be willing to put some cycles into if there is
> interest.
>
> -Kyle
>
> On Fri, Oct 14, 2016 at 2:15 PM, Ryan Merriman  > wrote:
>
> > +1 I like it.  Just to clarify, the scripts to run Storm topologies
> locally
> > in an IDE should be available independent of the environment running.  No
> > need for a separate build/image.
> >
> > On Fri, Oct 14, 2016 at 9:12 AM, Otto Fowler  >
> > wrote:
> >
> > > Going forward, the Demo env and data would have implications for
> testing
> > as
> > > well ( gold data sets ) etc.
> > >
> > > On October 14, 2016 at 09:52:07, Nick Allen (n...@nickallen.org
> ) wrote:
> > >
> > > I think based on everyone's input so far, we're describing 4 different
> > > builds/images/tools that would each be intended to run on a standard
> > > Mac/Linux/Windows laptop.
> > >
> > > Full Dev - A development environment that performs a full end-to-end
> > > deployment of Metron. This is intended for developers working with
> > > sensors, deployments, or validating how all Metron components interact
> > with
> > > one another.
> > >
> > >
> > > - Starts from base Linux image
> > > - Installs Hadoop-y components
> > > - Installs Metron
> > > - Installs Sensors
> > > - Nothing started by default
> > >
> > > Quick Dev - An environment intended for the developer focusing on the
> > > streaming components of Metron; parsing, enrichment, and indexing.
> > >
> > >
> > > - Starts from base image of Linux + Hadoop-y components
> > > - Installs Metron
> > > - Installs "data generator" spouts
> > > - Does not install sensors
> > > - Nothing started by default
> > >
> > > Demo - An environment intended to introduce new users to Metron. The
> > > environment should go from nothing to plenty of data in the Metron
> > > dashboard in as little "boot" time as possible.
> > >
> > >
> > > - Starts from a base image including Linux + Hadoop-y + Metron + Data
> > > Generator Spouts pre-installed
> > > - Pre-load Elasticsearch indices so the user has plenty of data to
> > > investigate in the dashboard
> > > - Does not install sensors
> > > - Everything started by default
> > >
> > > Storm Local Cluster - Otto suggested some scripts/tooling to make it
> easy
> > > to launch the core topologies on a local Storm cluster running on the
> > host
> > > OS.
> > >
> > >
> > > I'd be interested to hear if this works for everyone and how this might
> > > play into the Ambari mpack + RPM based deployment scheme.
> > >
> > >
> > > On Fri, Oct 14, 2016 at 1:45 AM, Michael Miklavcic <
> > > michael.miklav...@gmail.com > wrote:
> > >
> > > > I think this may have come up in another PR already (have to look for
> > > it).
> > > > But maybe we could maintain our flexibility in quick-dev by
> installing
> > > the
> > > > sensors and not starting them until we need them. I think it's useful
> > to
> > > > have a quick "genuine" e2e testing environment that doesn't require
> > > running
> > > > through a full install. I'm also not opposed to extracting the
> > > integration
> > > > test functionality into general purpose data generators.
> > > >
> > > > On Thu, Oct 13, 2016 at 8:31 PM, Nick Allen  >
> > wrote:
> > > >
> > > > > To Jon's point, I think it would be useful to have a Demo box that
> > uses
> > > > > generators to produce 3 or 4 types of telemetry that shows up in
> the
> > > > Metron
> > > > > Dashboard. This box would be different from Quick-Dev in that
> > > everything
> > > > > starts automatically, so that a user just has to launch it and the
> > > should
> > > > > start seeing data in the Metron Dashboard right away. In fact, we
> > could
> > > > > even pre-load the Elasticsearch indices so that the user has more
> of
> > a
> > > > > history to mine when using the Demo box.
> > > > >
> > > > > On Thu, Oct 13, 2016 at 2:04 PM, zeo...@gmail.com <
> zeo...@gmail.com >
> > > > > wrote:
> > > > >
> > > > > > +1 Ryan and Otto's comments.
> > > > > >
> > > > > > I also strongly think we need to make a demo environment easier,
> > but
> > > > that
> > > > > > should be different than quick-dev.
> > > > > >
> > > > > > Jon
> > > > > >
> > > > > > On Thu, Oct 13, 2016 at 1:15 PM Otto Fowler <
> > ottobackwa...@gmail.com 
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > - create scripts/utilities to easily run a topology 

Re: [DISCUSS] Improving quick-dev

[Kyle Richardson]

Sorry, I'm a bit late to the party on this one :).

+1 on the four builds Nick described. Each would be useful and purpose
built.

I like the idea of using Docker, especially for local development and quick
testing. Has anyone explored this? I'm envisioning very specific containers
so you could spin up only the components you're actively working on.
Certainly something I would be willing to put some cycles into if there is
interest.

-Kyle

On Fri, Oct 14, 2016 at 2:15 PM, Ryan Merriman  wrote:

> +1 I like it.  Just to clarify, the scripts to run Storm topologies locally
> in an IDE should be available independent of the environment running.  No
> need for a separate build/image.
>
> On Fri, Oct 14, 2016 at 9:12 AM, Otto Fowler 
> wrote:
>
> > Going forward, the Demo env and data would have implications for testing
> as
> > well ( gold data sets ) etc.
> >
> > On October 14, 2016 at 09:52:07, Nick Allen (n...@nickallen.org) wrote:
> >
> > I think based on everyone's input so far, we're describing 4 different
> > builds/images/tools that would each be intended to run on a standard
> > Mac/Linux/Windows laptop.
> >
> > Full Dev - A development environment that performs a full end-to-end
> > deployment of Metron. This is intended for developers working with
> > sensors, deployments, or validating how all Metron components interact
> with
> > one another.
> >
> >
> > - Starts from base Linux image
> > - Installs Hadoop-y components
> > - Installs Metron
> > - Installs Sensors
> > - Nothing started by default
> >
> > Quick Dev - An environment intended for the developer focusing on the
> > streaming components of Metron; parsing, enrichment, and indexing.
> >
> >
> > - Starts from base image of Linux + Hadoop-y components
> > - Installs Metron
> > - Installs "data generator" spouts
> > - Does not install sensors
> > - Nothing started by default
> >
> > Demo - An environment intended to introduce new users to Metron. The
> > environment should go from nothing to plenty of data in the Metron
> > dashboard in as little "boot" time as possible.
> >
> >
> > - Starts from a base image including Linux + Hadoop-y + Metron + Data
> > Generator Spouts pre-installed
> > - Pre-load Elasticsearch indices so the user has plenty of data to
> > investigate in the dashboard
> > - Does not install sensors
> > - Everything started by default
> >
> > Storm Local Cluster - Otto suggested some scripts/tooling to make it easy
> > to launch the core topologies on a local Storm cluster running on the
> host
> > OS.
> >
> >
> > I'd be interested to hear if this works for everyone and how this might
> > play into the Ambari mpack + RPM based deployment scheme.
> >
> >
> > On Fri, Oct 14, 2016 at 1:45 AM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > I think this may have come up in another PR already (have to look for
> > it).
> > > But maybe we could maintain our flexibility in quick-dev by installing
> > the
> > > sensors and not starting them until we need them. I think it's useful
> to
> > > have a quick "genuine" e2e testing environment that doesn't require
> > running
> > > through a full install. I'm also not opposed to extracting the
> > integration
> > > test functionality into general purpose data generators.
> > >
> > > On Thu, Oct 13, 2016 at 8:31 PM, Nick Allen 
> wrote:
> > >
> > > > To Jon's point, I think it would be useful to have a Demo box that
> uses
> > > > generators to produce 3 or 4 types of telemetry that shows up in the
> > > Metron
> > > > Dashboard. This box would be different from Quick-Dev in that
> > everything
> > > > starts automatically, so that a user just has to launch it and the
> > should
> > > > start seeing data in the Metron Dashboard right away. In fact, we
> could
> > > > even pre-load the Elasticsearch indices so that the user has more of
> a
> > > > history to mine when using the Demo box.
> > > >
> > > > On Thu, Oct 13, 2016 at 2:04 PM, zeo...@gmail.com 
> > > > wrote:
> > > >
> > > > > +1 Ryan and Otto's comments.
> > > > >
> > > > > I also strongly think we need to make a demo environment easier,
> but
> > > that
> > > > > should be different than quick-dev.
> > > > >
> > > > > Jon
> > > > >
> > > > > On Thu, Oct 13, 2016 at 1:15 PM Otto Fowler <
> ottobackwa...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > - create scripts/utilities to easily run a topology locally in an
> > IDE
> > > > > > instead of in the VM
> > > > > >
> > > > > >
> > > > > >  THIS.
> > > > > >
> > > > > >
> > > > > > On October 13, 2016 at 12:36:45, Ryan Merriman (
> > merrim...@gmail.com)
> >
> > > > > > wrote:
> > > > > >
> > > > > > Working with the quick-dev vagrant VM recently left a lot to be
> > > > desired.
> > > > > > All forthcoming comments are made under the assumption that this
> VM
> > > is
> > > > > > intended for development purposes. If that is not true, I think
> we
> > > > 

Re: [MENTORS] Release Maturity

[Kyle Richardson]

I'm +1 on a meeting to discuss the backlog and would suggest, to be
considered, each JIRA should have a clear description. I think that 72
hours is good for final adds to an upcoming release.

I personally like the idea of having more visibility on which JIRAs are "up
next" to help me figure out where contributions would be most valuable.

-Kyle

On Mon, Oct 17, 2016 at 1:50 PM, zeo...@gmail.com  wrote:

> That's more aggressive than I would have initially suggested, but I would
> be on board with that sort of a meeting.  Interested to see how others
> feel.
>
> Jon
>
> On Mon, Oct 17, 2016 at 1:40 PM James Sirota  wrote:
>
> > Fair criticism.  Would you like to call a recurring meeting where PPMC
> and
> > community can get together and go through the Jira backlog?  We can then
> > have the opportunity to triage the Jiras.  Do you think once a month
> should
> > be sufficient + 72 hours prior to the release to verify all desired Jiras
> > are in?
> >
> > 17.10.2016, 10:01, "zeo...@gmail.com" :
> > > I think that grooming the JIRA backlog at each release provides a good
> > > method for new users or users less integrated into Metron development
> to
> > > understand the roadmap of the project by perusing the backlog. I feel
> > > somewhat aware of the state of the Metron project but often have
> > questions
> > > about how prioritized certain issues are for development. I think the
> > > easiest way to illustrate this gap are in these
> > > <
> > https://issues.apache.org/jira/browse/METRON-170?jql=
> project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
> 20fixVersion%20%3D%200.2.1BETA%20ORDER%20BY%20priority%20DESC
> > >
> > > two
> > > <
> > https://issues.apache.org/jira/browse/METRON-469?jql=
> project%20%3D%20METRON%20AND%20resolution%20%3D%20Unresolved%20AND%
> 20fixVersion%20is%20EMPTY%20ORDER%20BY%20priority%20DESC
> > >
> > > links, where Metron currently has 45 unresolved issues slated for
> > 0.2.1BETA
> > > (?!?) and 71 unresolved issues that are unscheduled.
> > >
> > > I'd also like to see a comprehensive update of documentation on the
> wiki
> > >  (or
> > > wherever it lives). Heck, TP2 isn't even on the releases page
> > > , let
> alone
> > > 0.2.1 and other related details/changes.
> > >
> > > I'd be more than happy to help with either of those efforts, as
> > applicable.
> > >
> > > Jon
> > >
> > > On Mon, Oct 17, 2016 at 11:39 AM Casey Stella 
> > wrote:
> > >
> > >>  Hi Everyone,
> > >>
> > >>  I'd like to get a bit more systematic about how we release and I
> wanted
> > >>  some clarification and advice about suggested release process.
> > >>
> > >>  The last release, we
> > >>
> > >> - opened up the release via an announce thread that gave people
> the
> > >> opportunity to object and add JIRAs they felt were important to be
> > >> considered for the release
> > >> - made a release branch in git
> > >> - made a release candidate tag
> > >> - sent out the release candidate for a vote
> > >> - when passed, sent the release candidate for a vote in general
> > >>
> > >>  A couple of questions:
> > >>
> > >> - Is 72 hours sufficient for people to suggest JIRAs that need to
> > get in
> > >> for the release?
> > >> - What we did not do is have the JIRA backlog groomed and have
> JIRAs
> > >> assigned to releases beyond the current release. This would make
> it
> > >>  easier
> > >> for people to find JIRAs that they want in. Is that a sensible
> > >> prerequisite for the release or is that overkill?
> > >> - Are there best practices that successful projects of our
> > >> maturity-level do that we are not doing around release?
> > >>
> > >>  Casey
> > > --
> > >
> > > Jon
> >
> > ---
> > Thank you,
> >
> > James Sirota
> > PPMC- Apache Metron (Incubating)
> > jsirota AT apache DOT org
> >
> --
>
> Jon
>

[GitHub] incubator-metron issue #308: Metron-498 Grok patterns are now read from zook...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/incubator-metron/pull/308
  
Can't argue with you there, that is a better name.  Thanks for all the 
great suggestions @mattf-horton.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #308: Metron-498 Grok patterns are now read from zook...

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/308
  
Ah, that makes sense -- although I would have called it 
"timestampFieldName" :-)

+1.  I think it's ready to go, and a great piece of work!
Thanks for all your patience during the review process.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r84110876
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
 ---
@@ -181,7 +185,8 @@ public void declareOutputFields(OutputFieldsDeclarer 
declarer) {
   @Override
   public void updateConfig(String path, byte[] data) throws IOException {
 super.updateConfig(path, data);
-if (path.startsWith(ConfigurationType.PARSER.getZookeeperRoot() + "/" 
+ getSensorType())) {
+String pathWithoutTrailingSlash = path.replaceAll("/+$", "");
+if 
(pathWithoutTrailingSlash.equals(ConfigurationType.PARSER.getZookeeperRoot() + 
"/" + getSensorType())) {
--- End diff --

Ok, if sensorType is known to always be a leaf node in ZK, that's 
sufficient.  Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #305: METRON-403 Bro Elasticsearch index item fails w...

[james-sirota]

Github user james-sirota commented on the issue:

https://github.com/apache/incubator-metron/pull/305
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #315: METRON-464: Force co-location of all Met...

[dlyle65535]

GitHub user dlyle65535 opened a pull request:

https://github.com/apache/incubator-metron/pull/315

METRON-464: Force co-location of all Metron components

Now the Ambari Add Service will show an error in the case that:

a) The Metron Parsers service is not deployed with the Storm Supervisor and 
Kafka Broker.
b) Any of the other Metron services aren't installed to the host containing 
Metron Parsers.

Tested on Docker cluster following procedure here:[ Metron 
MPack](https://www.evernote.com/l/AhLFVR-9CsFIYYnOnF43BlxSsT4F856qwaY)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dlyle65535/incubator-metron METRON-464

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/315.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #315


commit b40f0d649f461a9adb82495e912f7cc0558d8503
Author: David Lyle 
Date:   2016-10-17T23:28:46Z

METRON-464: Force co-location of all Metron components




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #310: METRON-495: Upgrade Storm to 1.0.x

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/310
  
I spent some time on this yesterday.  I do not have a lot of Kafka/ZK/Storm 
experience - but I was hoping for something to stand out 
Unfortunately that was not the case.  Can you share how you setup intellij 
for scala?  I'm seeing intellij not finding the wrapper classes.
Also - if you have any more information on the failing tests or guesses as 
to their cause that would help.

Do we know if we should have a newer version of curator to go with the 
newer kafka/storm?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser

[kylerichardson]

Github user kylerichardson commented on the issue:

https://github.com/apache/incubator-metron/pull/276
  
Whew, got the CI build to finally pass. All integration and unit tests are 
passing. I've also re-testing in the single node vm environment I described 
above.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Build Log Usability

[David Lyle]

Hi Otto,

Good point. One of the practices I'd like to see is to suppress any
exception/error output in tests when it's expected.

Thanks!

-D...


On Wed, Oct 19, 2016 at 8:18 AM, Otto Fowler 
wrote:

> The Metron build logs contain a great deal of information, including a
> large number of warnings / exceptions that do not result in test failures
> but still fill the log.  This makes troubleshooting test/build issues
> difficult.  For example - the travis ci log for PR #276’s recent failure
> doesn’t even have the final errors because the log is too long.
>
> What are some things that can be done to make the situation better?
>
>
>

[DISCUSS] Build Log Usability

[Otto Fowler]

The Metron build logs contain a great deal of information, including a large 
number of warnings / exceptions that do not result in test failures but still 
fill the log.  This makes troubleshooting test/build issues difficult.  For 
example - the travis ci log for PR #276’s recent failure doesn’t even have the 
final errors because the log is too long.

What are some things that can be done to make the situation better?

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

[kylerichardson]

GitHub user kylerichardson reopened a pull request:

https://github.com/apache/incubator-metron/pull/276

METRON-363 Fix Cisco ASA Parser

I've rewritten the ASA parser which can be extended, as needed, to new ASA 
message types by editing the bundled asa patterns file and the static map used 
for grok patterns in the code. I've also tried to make it easier to deploy the 
asa topology by including zookeeper config files and creating the kafka topic 
during metron install. Sample data is also included for integration testing.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kylerichardson/incubator-metron METRON-363

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/276.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #276


commit 5be7c60448f73fcc72c81451a67ef1e40fd29793
Author: kylerichardson 
Date:   2016-08-16T01:12:42Z

Initial rewrite of Cisco ASA parser

Summary of changes:
- Complete rewrite of ASA parser including new test suite
- ZK configurations for ease of topology deployment (parser and enrichment)
- Add field constant for original_string in metron-common
- Minor changes to ASA patterns file for
  (1) Syslog severity/facility capture
  (2) Interface capture on CISCOFW106006_106007_106010
- Updates to various POMs to allow easier validation of logging during unit 
testing
  (1) Exclusions for slf4j-log4j12 on various dependencies for 
metron-parsers and metron-integration-test
  (2) Explicit dependency on slf4j-api for metron-parsers
  (3) Test dependency on slf4j-simple for metron-parsers

commit c87e6edaf0e308be9f417e07016508f87067ae0c
Author: kylerichardson 
Date:   2016-09-20T02:33:09Z

METRON-363 Reworked parser to handle nulls and field validation

Includes the following:
- Static map for ASA message patterns (vs pattern discovery)
- Minor changes to ASA patterns file
- Broke out common syslog parsing elements
- Broke out reusable field validations

commit a8c4903dd0bcac18e15c98aca7264dce1c455bee
Author: kylerichardson 
Date:   2016-09-27T00:30:16Z

METRON-363 Add integration test and sample data

Includes the following:
- Extend BasicParser
- Handle both types of syslog timestamps (with and without year)
- Include integration test and supporting sample data

commit 011d389bdf43f1790384dbcd13ec7da148c53ef2
Author: kylerichardson 
Date:   2016-09-27T00:40:51Z

METRON-363 Add license and kafka topic

commit 04a936d75cf782254105993b2804912b4659257a
Author: kylerichardson 
Date:   2016-09-28T00:29:21Z

METRON-363 Adjust log level

commit abd7fb92fe4c38530e10141d0aba6bd07a335ae8
Author: kylerichardson 
Date:   2016-10-08T01:11:22Z

METRON-363 Enhance logging, remove unused code

commit a885ecc762a8d5296d7c7ebfe7600c910ce3478b
Author: kylerichardson 
Date:   2016-10-11T17:40:25Z

METRON-363 Refactored and enhanced based on feedback

Changes include:
(1) New/additional unit tests
(2) Reworked Syslog Timestamp (no year) logic
(3) Enhanced error checking and logging (introduced new ParseException)

commit fb6ed83eab8704607dc75c37982b0f98b819047d
Author: kylerichardson 
Date:   2016-10-12T13:54:54Z

METRON-363 Default to UTC in zookeeper config

commit d7d327a3b03584fd3d03d4f6468d54c15786bda7
Author: kylerichardson 
Date:   2016-10-13T02:10:14Z

METRON-363 Update tests

commit 4e3cba6682eaf3130325d4c27bf32240ad7a0a92
Author: kylerichardson 
Date:   2016-10-18T00:33:34Z

METRON-363 Refactor to add Clock dependency for testing

commit db8686615533470e8a3273ee268f2eb0efb4999c
Author: kylerichardson 
Date:   2016-10-18T01:15:29Z

METRON-363 Add tests for back dating RFC3164 timestamps




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

[kylerichardson]

Github user kylerichardson closed the pull request at:

https://github.com/apache/incubator-metron/pull/276


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---