date:20161001

Re: Complete steps to add a new parser

2016-10-01 Thread Otto Fowler

I have been able to add a new parser to the the deployment, and have the
cluster fully deploy successfully.  After I was able to push data to kafka
from HDF and get it all indexed.

Unlike quick dev and full, no problems getting the storm ports correct
after deployment to my small cluster config.

It looks to me that the steps I took to integrate the parser worked, but I
still may have missed something.  One thing that I know I missed was
modifying the dashboard - adding the saved searches and integrating them
with the visualizations.

Here is a gist off a patch of my changes.  The patch in the gist has been
modified - so I don’t think it will apply for you.  I removed proprietary
field names ( the stellar config, the enrichment hosts, es index template ).

I hope what is there is enough for you to verify, correct what I have done.

https://gist.github.com/ottobackwards/1c03299bb85a2d3b266c74202df71f84

On September 27, 2016 at 13:42:51, Otto Fowler (ottobackwa...@gmail.com)
wrote:

Thanks Nick,
That is some of the stuff that I have found trying to track down the deploy
bits of the existing parsers, but I don’t want to miss anything, so I’d
like some guidence.  Right now, I’m OK with doing it all inside the metron
ansible base.  I expect that once I get it working and wrap my head around
it I’ll have some ideas that I’ll float around improving this area, some
use cases to propose that would possibly be external to the main
deployment, or additive. First thing first is understanding all the points
in doing it the hard way ;)

-- 

Sent with Airmail

On September 27, 2016 at 12:41:10, Nick Allen (n...@nickallen.org) wrote:

Hi Otto -

I would agree with you. We do not have documentation that describes how to
'permanently install' a new parser.. Your contribution would be highly
appreciated in this area.

With the Ansible-based deployment of today, most likely you will have to
touch some of Metron's Ansible source code. An alternative would be to
mimic portions of Metron's deployment code, and manage that in its own
project, which would deploy your new parser. But of course, if we can find
ways to make this task easier, we will.

You may not have to touch each of these areas, but they at least will
provide you with a better understanding of how everything is stitched
together.

*Monit *- The Monit integration lives in `metron-deployment/roles/monit`.
You can follow the pattern of
metron-deployment/roles/monit/templates/monit/parsers.monit to add your own
parser definition to Monit.

*Parsers* - The start script in
`metron-platform/metron-parsers/src/main/scripts/start_parser_topology.sh`
will give you good hooks into how each of the parsers are started.

*Setup* - There are various setup tasks for the streaming functionality
that live under `metron-deployment/roles/metron_streaming`. To understand
that process, start at `tasks/main.yml`.

I probably missed something, but let me know if you have questions.

On Tue, Sep 27, 2016 at 12:17 PM, Otto Fowler 
wrote:

> My wish, is that when I do an ansible-playbook -v -i {my configuration}
> metron_full_install.yml to my cluster - or do the full_dev-> vagrant that
> my parser / topology is deployed, started and monitored the same way as
the
> current bro, snort, and yaf parsers are.
>
> I might be misunderstanding something however. I seems to me that all the
> examples of adding other parsers are temporary and not permanent because
> they do not have the full deployment, kind of push the config and run the
> script and you are going. Am I missing something? Would the squid sample
> steps result in a parser topology that would survive restarts / reboots
> etc?
>
> On September 27, 2016 at 12:06:44, James Sirota (jsir...@apache.org)
> wrote:
>
> Just so I completely understand what you are asking for...you want to know
> how to create a new parser topology with the JSON parser and plug it into
> Monit so you can monitor and restart it on demand?
>
> 27.09.2016, 09:03, "Otto Fowler" :
> > Thanks James,
> >
> > I want to deploy an instance of the JSONMapParser into my POC cluster
and
> vagrant. I’m trying to work out exactly how to add a new configured parser
> instance to the deployment. I think these instructions would be a good
> extension to the squid stuff that is already there. If I could get that
> going and add a new parser all the way through, then maybe I can
contribute
> something in that area. The ability to do this will also enable some of
> the other work you mentioned.
> >
> > On September 27, 2016 at 11:51:41, James Sirota (jsir...@apache.org)
> wrote:
> >
> >> There are three types of parsers you can have currently. Our preferred
> way is to use Grok parser. The only thing you need to do there is to
define
> your Grok statement and the parser will uptake it and do the rest. That is
> what most of our documentation reflect. The second type of parser that we
> have is a java parser, where you

Re: [DISCUSS] Dockerize Metron

2016-10-01 Thread Michael Miklavcic

I really like this

On Oct 1, 2016 5:03 PM, "Ryan Merriman"  wrote:

> I don't think this will be an issue.  The remote debugging strategy should
> work if you want to go that route but I would just run the topology in
> local mode (that's how the integration tests work now).  The Storm topology
> continues  to runs locally in your IDE while all the other components that
> were in-memory components (HBase, Kafka, etc) now run in Docker
> containers.  We can continue to leverage
> https://github.com/apache/incubator-metron/blob/master/
> metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/
> integration/components/ParserTopologyComponent.java
> for parser topologies and the --local option for flux-based topologies.
>
> On Sat, Oct 1, 2016 at 12:28 PM, Nick Allen  wrote:
>
> > It's definitely possible to debug in a Docker environment using a remote
> > debugger.  I remember reading something along these lines a few weeks
> ago.
> >
> > https://blog.docker.com/2016/09/java-development-using-docker/
> >
> > On Sat, Oct 1, 2016 at 1:15 PM, Casey Stella  wrote:
> >
> > > I agree with that concern. Being able to debug a running topology has
> > > helped in so many circumstances. Not sure how to accomplish this in a
> > > dockerized environment.
> > >
> > > On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic <
> > > michael.miklav...@gmail.com>
> > > wrote:
> > >
> > > > Any ideas on how debugging works when leveraging Docker? In spite of
> > the
> > > > classpath troubles, one of the benefits of the current single-JVM
> > > approach
> > > > is that you can easily debug, set break points, etc within an IDE. Is
> > > that
> > > > still doable? Seems like there might need to be some remote debugging
> > > magic
> > > > done to accomplish this.
> > > >
> > > > On Fri, Sep 30, 2016 at 6:39 PM, James Sirota 
> > > wrote:
> > > >
> > > > > I agree with making an effort to create containers.  I suggest
> doing
> > it
> > > > on
> > > > > a feature branch until we are feature complete and are able to
> > migrate
> > > > our
> > > > > integration tests into a dockerized environment.
> > > > >
> > > > > 30.09.2016, 14:52, "Nick Allen" :
> > > > > >  Relieve dependency version conflict issues? Umm, yes please.
> I'll
> > > > take a
> > > > > >  second helping of that too.
> > > > > >
> > > > > >  On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
> > merrim...@gmail.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > >>   I would like to open up a discussion around creating Docker
> > images
> > > > for
> > > > > >>   Metron. Having this available would provide a leaner
> alternative
> > > to
> > > > > the
> > > > > >>   ansible/vagrant environment for development tesing (and even
> > > demoing
> > > > > or
> > > > > >>   exploring features). It could also relieve some of the
> > dependency
> > > > > version
> > > > > >>   conflict issues that we've been experiencing when running
> > > > integration
> > > > > tests
> > > > > >>   in a single JVM.
> > > > > >>
> > > > > >>   I would suggest the initial version be intended only for
> > > development
> > > > > and
> > > > > >>   testing purposes. The general approach could be to create an
> > image
> > > > for
> > > > > >>   each service we depend on and use something like Docker
> compose
> > to
> > > > > package
> > > > > >>   them together. A Dockerfile would either install the service
> > from
> > > > > scratch
> > > > > >>   or extend a community image then add any Metron related
> > > dependencies
> > > > > or
> > > > > >>   configurations on top. The metron-deployment project code
> could
> > be
> > > > > used as
> > > > > >>   a guide.
> > > > > >>
> > > > > >>   I would like to see these images added initially to support
> > > > > development and
> > > > > >>   testing:
> > > > > >>
> > > > > >>  - Kafka with topics preconfigured
> > > > > >>  - Storm with metron topology assets installed
> > > > > >>  - Zookeeper with paths created and sample configs loaded
> > > > > >>  - HBase with sample enrichments and threat intel loaded
> > > > > >>  - Elasticsearch configured for Metron
> > > > > >>  - MySQL with databases/tables/users created and geo data
> > loaded
> > > > > >>
> > > > > >>   Other images that could also be useful:
> > > > > >>
> > > > > >>  - Images for each sensor
> > > > > >>  - Ambari?
> > > > > >>  - Solr
> > > > > >>
> > > > > >>   Looking forward to hearing what everyone thinks.
> > > > > >>
> > > > > >>   Ryan Merriman
> > > > > >
> > > > > >  --
> > > > > >  Nick Allen 
> > > > >
> > > > >
> > > > > ---
> > > > > Thank you,
> > > > >
> > > > > James Sirota
> > > > > PPMC- Apache Metron (Incubating)
> > > > > jsirota AT apache DOT org
> > > > >
> > > > > 30.09.2016, 14:52, "Nick Allen" :
> > > > > > Relieve dependency version conflict issues? Umm,

Re: [DISCUSS] Dockerize Metron

2016-10-01 Thread P. Taylor Goetz

At least on the Storm side, remote debugging can be problematic. If you pause 
execution of a worker process, it will stop heart beating and supervisor will 
kill it. Just something to be aware of.

I think we might have a flag to disable that behavior, but I'd have to check.

I usually use local mode for debugging.

-Taylor

> On Oct 1, 2016, at 5:03 PM, Ryan Merriman  wrote:
> 
> I don't think this will be an issue.  The remote debugging strategy should
> work if you want to go that route but I would just run the topology in
> local mode (that's how the integration tests work now).  The Storm topology
> continues  to runs locally in your IDE while all the other components that
> were in-memory components (HBase, Kafka, etc) now run in Docker
> containers.  We can continue to leverage
> https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/components/ParserTopologyComponent.java
> for parser topologies and the --local option for flux-based topologies.
> 
>> On Sat, Oct 1, 2016 at 12:28 PM, Nick Allen  wrote:
>> 
>> It's definitely possible to debug in a Docker environment using a remote
>> debugger.  I remember reading something along these lines a few weeks ago.
>> 
>> https://blog.docker.com/2016/09/java-development-using-docker/
>> 
>>> On Sat, Oct 1, 2016 at 1:15 PM, Casey Stella  wrote:
>>> 
>>> I agree with that concern. Being able to debug a running topology has
>>> helped in so many circumstances. Not sure how to accomplish this in a
>>> dockerized environment.
>>> 
>>> On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic <
>>> michael.miklav...@gmail.com>
>>> wrote:
>>> 
 Any ideas on how debugging works when leveraging Docker? In spite of
>> the
 classpath troubles, one of the benefits of the current single-JVM
>>> approach
 is that you can easily debug, set break points, etc within an IDE. Is
>>> that
 still doable? Seems like there might need to be some remote debugging
>>> magic
 done to accomplish this.
 
> On Fri, Sep 30, 2016 at 6:39 PM, James Sirota 
 wrote:
 
> I agree with making an effort to create containers.  I suggest doing
>> it
 on
> a feature branch until we are feature complete and are able to
>> migrate
 our
> integration tests into a dockerized environment.
> 
> 30.09.2016, 14:52, "Nick Allen" :
>> Relieve dependency version conflict issues? Umm, yes please. I'll
 take a
>> second helping of that too.
>> 
>> On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
>> merrim...@gmail.com
 
> wrote:
>> 
>>>  I would like to open up a discussion around creating Docker
>> images
 for
>>>  Metron. Having this available would provide a leaner alternative
>>> to
> the
>>>  ansible/vagrant environment for development tesing (and even
>>> demoing
> or
>>>  exploring features). It could also relieve some of the
>> dependency
> version
>>>  conflict issues that we've been experiencing when running
 integration
> tests
>>>  in a single JVM.
>>> 
>>>  I would suggest the initial version be intended only for
>>> development
> and
>>>  testing purposes. The general approach could be to create an
>> image
 for
>>>  each service we depend on and use something like Docker compose
>> to
> package
>>>  them together. A Dockerfile would either install the service
>> from
> scratch
>>>  or extend a community image then add any Metron related
>>> dependencies
> or
>>>  configurations on top. The metron-deployment project code could
>> be
> used as
>>>  a guide.
>>> 
>>>  I would like to see these images added initially to support
> development and
>>>  testing:
>>> 
>>> - Kafka with topics preconfigured
>>> - Storm with metron topology assets installed
>>> - Zookeeper with paths created and sample configs loaded
>>> - HBase with sample enrichments and threat intel loaded
>>> - Elasticsearch configured for Metron
>>> - MySQL with databases/tables/users created and geo data
>> loaded
>>> 
>>>  Other images that could also be useful:
>>> 
>>> - Images for each sensor
>>> - Ambari?
>>> - Solr
>>> 
>>>  Looking forward to hearing what everyone thinks.
>>> 
>>>  Ryan Merriman
>> 
>> --
>> Nick Allen 
> 
> 
> ---
> Thank you,
> 
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
> 
> 30.09.2016, 14:52, "Nick Allen" :
>> Relieve dependency version conflict issues? Umm, yes please. I'll
>>> take
 a
>> second helping of that too.
>> 
>> On Fri, Sep 30, 2016 at 5:00 PM, Ryan

Re: [DISCUSS] Dockerize Metron

2016-10-01 Thread Casey Stella

Oh yeah I like that strategy. It lets us isolate some of our more
"challenging" dependencies in the tests (e.g. Hbase and storm) and still
debug in the ide.
On Sat, Oct 1, 2016 at 17:03 Ryan Merriman  wrote:

> I don't think this will be an issue.  The remote debugging strategy should
> work if you want to go that route but I would just run the topology in
> local mode (that's how the integration tests work now).  The Storm topology
> continues  to runs locally in your IDE while all the other components that
> were in-memory components (HBase, Kafka, etc) now run in Docker
> containers.  We can continue to leverage
>
> https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/components/ParserTopologyComponent.java
> for parser topologies and the --local option for flux-based topologies.
>
> On Sat, Oct 1, 2016 at 12:28 PM, Nick Allen  wrote:
>
> > It's definitely possible to debug in a Docker environment using a remote
> > debugger.  I remember reading something along these lines a few weeks
> ago.
> >
> > https://blog.docker.com/2016/09/java-development-using-docker/
> >
> > On Sat, Oct 1, 2016 at 1:15 PM, Casey Stella  wrote:
> >
> > > I agree with that concern. Being able to debug a running topology has
> > > helped in so many circumstances. Not sure how to accomplish this in a
> > > dockerized environment.
> > >
> > > On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic <
> > > michael.miklav...@gmail.com>
> > > wrote:
> > >
> > > > Any ideas on how debugging works when leveraging Docker? In spite of
> > the
> > > > classpath troubles, one of the benefits of the current single-JVM
> > > approach
> > > > is that you can easily debug, set break points, etc within an IDE. Is
> > > that
> > > > still doable? Seems like there might need to be some remote debugging
> > > magic
> > > > done to accomplish this.
> > > >
> > > > On Fri, Sep 30, 2016 at 6:39 PM, James Sirota 
> > > wrote:
> > > >
> > > > > I agree with making an effort to create containers.  I suggest
> doing
> > it
> > > > on
> > > > > a feature branch until we are feature complete and are able to
> > migrate
> > > > our
> > > > > integration tests into a dockerized environment.
> > > > >
> > > > > 30.09.2016, 14:52, "Nick Allen" :
> > > > > >  Relieve dependency version conflict issues? Umm, yes please.
> I'll
> > > > take a
> > > > > >  second helping of that too.
> > > > > >
> > > > > >  On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
> > merrim...@gmail.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > >>   I would like to open up a discussion around creating Docker
> > images
> > > > for
> > > > > >>   Metron. Having this available would provide a leaner
> alternative
> > > to
> > > > > the
> > > > > >>   ansible/vagrant environment for development tesing (and even
> > > demoing
> > > > > or
> > > > > >>   exploring features). It could also relieve some of the
> > dependency
> > > > > version
> > > > > >>   conflict issues that we've been experiencing when running
> > > > integration
> > > > > tests
> > > > > >>   in a single JVM.
> > > > > >>
> > > > > >>   I would suggest the initial version be intended only for
> > > development
> > > > > and
> > > > > >>   testing purposes. The general approach could be to create an
> > image
> > > > for
> > > > > >>   each service we depend on and use something like Docker
> compose
> > to
> > > > > package
> > > > > >>   them together. A Dockerfile would either install the service
> > from
> > > > > scratch
> > > > > >>   or extend a community image then add any Metron related
> > > dependencies
> > > > > or
> > > > > >>   configurations on top. The metron-deployment project code
> could
> > be
> > > > > used as
> > > > > >>   a guide.
> > > > > >>
> > > > > >>   I would like to see these images added initially to support
> > > > > development and
> > > > > >>   testing:
> > > > > >>
> > > > > >>  - Kafka with topics preconfigured
> > > > > >>  - Storm with metron topology assets installed
> > > > > >>  - Zookeeper with paths created and sample configs loaded
> > > > > >>  - HBase with sample enrichments and threat intel loaded
> > > > > >>  - Elasticsearch configured for Metron
> > > > > >>  - MySQL with databases/tables/users created and geo data
> > loaded
> > > > > >>
> > > > > >>   Other images that could also be useful:
> > > > > >>
> > > > > >>  - Images for each sensor
> > > > > >>  - Ambari?
> > > > > >>  - Solr
> > > > > >>
> > > > > >>   Looking forward to hearing what everyone thinks.
> > > > > >>
> > > > > >>   Ryan Merriman
> > > > > >
> > > > > >  --
> > > > > >  Nick Allen 
> > > > >
> > > > >
> > > > > ---
> > > > > Thank you,
> > > > >
> > > > > James Sirota
> > > > > PPMC- Apache Metron (Incubating)
> > > > > jsirota AT apache DOT org
>

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

2016-10-01 Thread kylerichardson

Github user kylerichardson commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/276#discussion_r81454727
  
--- Diff: metron-deployment/roles/metron_kafka_topics/defaults/main.yml ---
@@ -21,6 +21,7 @@ topics_to_create:
   - { topic: "bro", num_partitions: 1, replication_factor: 1, 
retention_gb: 10 }
   - { topic: "yaf", num_partitions: 1, replication_factor: 1, 
retention_gb: 10 }
   - { topic: "snort",   num_partitions: 1, replication_factor: 1, 
retention_gb: 10 }
+  - { topic: "asa", num_partitions: 1, replication_factor: 1, 
retention_gb: 10 }
--- End diff --

That makes sense. I was thinking about building out the monit scripts, etc 
to make this as easy as possible for the user to deploy out-of-the-box, but 
that's a future PR. Is that something that would be valuable to folks? Either 
way, I can remove this from the current PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

2016-10-01 Thread kylerichardson

Github user kylerichardson commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/276#discussion_r81454659
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/BasicAsaParser.java
 ---
@@ -0,0 +1,165 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.asa;
+
+import com.google.common.collect.ImmutableMap;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.commons.validator.routines.InetAddressValidator;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.utils.FieldValidators;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.*;
+import java.util.*;
+
+public class BasicAsaParser extends BasicParser {
+
+protected static final Logger LOG = 
LoggerFactory.getLogger(BasicAsaParser.class);
+
+private Grok asaGrok;
+
+private static final InetAddressValidator ipValidator = 
InetAddressValidator.getInstance();
+
+private static final Map patternMap = 
ImmutableMap.builder()
+.put("ASA-2-106001", "CISCOFW106001")
+   .put("ASA-2-106006", "CISCOFW106006_106007_106010")
+   .put("ASA-2-106007", "CISCOFW106006_106007_106010")
+   .put("ASA-2-106010", "CISCOFW106006_106007_106010")
+   .put("ASA-3-106014", "CISCOFW106014")
+   .put("ASA-6-106015", "CISCOFW106015")
+   .put("ASA-1-106021", "CISCOFW106021")
+   .put("ASA-4-106023", "CISCOFW106023")
+   .put("ASA-5-106100", "CISCOFW106100")
+   .put("ASA-6-110002", "CISCOFW110002")
+   .put("ASA-6-302010", "CISCOFW302010")
+   .put("ASA-6-302013", "CISCOFW302013_302014_302015_302016")
+   .put("ASA-6-302014", "CISCOFW302013_302014_302015_302016")
+   .put("ASA-6-302015", "CISCOFW302013_302014_302015_302016")
+   .put("ASA-6-302016", "CISCOFW302013_302014_302015_302016")
+   .put("ASA-6-302020", "CISCOFW302020_302021")
+   .put("ASA-6-302021", "CISCOFW302020_302021")
+   .put("ASA-6-305011", "CISCOFW305011")
+   .put("ASA-3-313001", "CISCOFW313001_313004_313008")
+   .put("ASA-3-313004", "CISCOFW313001_313004_313008")
+   .put("ASA-3-313008", "CISCOFW313001_313004_313008")
+   .put("ASA-4-313005", "CISCOFW313005")
+   .put("ASA-4-402117", "CISCOFW402117")
+   .put("ASA-4-402119", "CISCOFW402119")
+   .put("ASA-4-419001", "CISCOFW419001")
+   .put("ASA-4-419002", "CISCOFW419002")
+   .put("ASA-4-54", "CISCOFW54")
+   .put("ASA-6-602303", "CISCOFW602303_602304")
+   .put("ASA-6-602304", "CISCOFW602303_602304")
+   .put("ASA-7-710001", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-7-710002", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-7-710003", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-7-710005", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-7-710006", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-6-713172", "CISCOFW713172")
+   .put("ASA-4-733100", "CISCOFW733100")
+   .put("ASA-6-305012", "CISCOFW305012")
+   .put("ASA-7-609001", "CISCOFW609001")
+   .put("ASA-7-609002", "CISCOFW609002")
+.put("ASA-5-713041", "CISCOFW713041")
+.build();
+
+@Override
+

Re: [DISCUSS] Dockerize Metron

2016-10-01 Thread Nick Allen

It's definitely possible to debug in a Docker environment using a remote
debugger.  I remember reading something along these lines a few weeks ago.

https://blog.docker.com/2016/09/java-development-using-docker/

On Sat, Oct 1, 2016 at 1:15 PM, Casey Stella  wrote:

> I agree with that concern. Being able to debug a running topology has
> helped in so many circumstances. Not sure how to accomplish this in a
> dockerized environment.
>
> On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic <
> michael.miklav...@gmail.com>
> wrote:
>
> > Any ideas on how debugging works when leveraging Docker? In spite of the
> > classpath troubles, one of the benefits of the current single-JVM
> approach
> > is that you can easily debug, set break points, etc within an IDE. Is
> that
> > still doable? Seems like there might need to be some remote debugging
> magic
> > done to accomplish this.
> >
> > On Fri, Sep 30, 2016 at 6:39 PM, James Sirota 
> wrote:
> >
> > > I agree with making an effort to create containers.  I suggest doing it
> > on
> > > a feature branch until we are feature complete and are able to migrate
> > our
> > > integration tests into a dockerized environment.
> > >
> > > 30.09.2016, 14:52, "Nick Allen" :
> > > >  Relieve dependency version conflict issues? Umm, yes please. I'll
> > take a
> > > >  second helping of that too.
> > > >
> > > >  On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman  >
> > > wrote:
> > > >
> > > >>   I would like to open up a discussion around creating Docker images
> > for
> > > >>   Metron. Having this available would provide a leaner alternative
> to
> > > the
> > > >>   ansible/vagrant environment for development tesing (and even
> demoing
> > > or
> > > >>   exploring features). It could also relieve some of the dependency
> > > version
> > > >>   conflict issues that we've been experiencing when running
> > integration
> > > tests
> > > >>   in a single JVM.
> > > >>
> > > >>   I would suggest the initial version be intended only for
> development
> > > and
> > > >>   testing purposes. The general approach could be to create an image
> > for
> > > >>   each service we depend on and use something like Docker compose to
> > > package
> > > >>   them together. A Dockerfile would either install the service from
> > > scratch
> > > >>   or extend a community image then add any Metron related
> dependencies
> > > or
> > > >>   configurations on top. The metron-deployment project code could be
> > > used as
> > > >>   a guide.
> > > >>
> > > >>   I would like to see these images added initially to support
> > > development and
> > > >>   testing:
> > > >>
> > > >>  - Kafka with topics preconfigured
> > > >>  - Storm with metron topology assets installed
> > > >>  - Zookeeper with paths created and sample configs loaded
> > > >>  - HBase with sample enrichments and threat intel loaded
> > > >>  - Elasticsearch configured for Metron
> > > >>  - MySQL with databases/tables/users created and geo data loaded
> > > >>
> > > >>   Other images that could also be useful:
> > > >>
> > > >>  - Images for each sensor
> > > >>  - Ambari?
> > > >>  - Solr
> > > >>
> > > >>   Looking forward to hearing what everyone thinks.
> > > >>
> > > >>   Ryan Merriman
> > > >
> > > >  --
> > > >  Nick Allen 
> > >
> > >
> > > ---
> > > Thank you,
> > >
> > > James Sirota
> > > PPMC- Apache Metron (Incubating)
> > > jsirota AT apache DOT org
> > >
> > > 30.09.2016, 14:52, "Nick Allen" :
> > > > Relieve dependency version conflict issues? Umm, yes please. I'll
> take
> > a
> > > > second helping of that too.
> > > >
> > > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman 
> > > wrote:
> > > >
> > > >>  I would like to open up a discussion around creating Docker images
> > for
> > > >>  Metron. Having this available would provide a leaner alternative to
> > the
> > > >>  ansible/vagrant environment for development tesing (and even
> demoing
> > or
> > > >>  exploring features). It could also relieve some of the dependency
> > > version
> > > >>  conflict issues that we've been experiencing when running
> integration
> > > tests
> > > >>  in a single JVM.
> > > >>
> > > >>  I would suggest the initial version be intended only for
> development
> > > and
> > > >>  testing purposes. The general approach could be to create an image
> > for
> > > >>  each service we depend on and use something like Docker compose to
> > > package
> > > >>  them together. A Dockerfile would either install the service from
> > > scratch
> > > >>  or extend a community image then add any Metron related
> dependencies
> > or
> > > >>  configurations on top. The metron-deployment project code could be
> > > used as
> > > >>  a guide.
> > > >>
> > > >>  I would like to see these images added initially to support
> > > development and
> > > >>  testing:

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

2016-10-01 Thread kylerichardson

Github user kylerichardson commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/276#discussion_r81454500
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/BasicAsaParser.java
 ---
@@ -0,0 +1,165 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.asa;
+
+import com.google.common.collect.ImmutableMap;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.commons.validator.routines.InetAddressValidator;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.utils.FieldValidators;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.*;
+import java.util.*;
+
+public class BasicAsaParser extends BasicParser {
+
+protected static final Logger LOG = 
LoggerFactory.getLogger(BasicAsaParser.class);
+
+private Grok asaGrok;
+
+private static final InetAddressValidator ipValidator = 
InetAddressValidator.getInstance();
+
+private static final Map patternMap = 
ImmutableMap.builder()
+.put("ASA-2-106001", "CISCOFW106001")
+   .put("ASA-2-106006", "CISCOFW106006_106007_106010")
+   .put("ASA-2-106007", "CISCOFW106006_106007_106010")
+   .put("ASA-2-106010", "CISCOFW106006_106007_106010")
+   .put("ASA-3-106014", "CISCOFW106014")
+   .put("ASA-6-106015", "CISCOFW106015")
+   .put("ASA-1-106021", "CISCOFW106021")
+   .put("ASA-4-106023", "CISCOFW106023")
+   .put("ASA-5-106100", "CISCOFW106100")
+   .put("ASA-6-110002", "CISCOFW110002")
+   .put("ASA-6-302010", "CISCOFW302010")
+   .put("ASA-6-302013", "CISCOFW302013_302014_302015_302016")
+   .put("ASA-6-302014", "CISCOFW302013_302014_302015_302016")
+   .put("ASA-6-302015", "CISCOFW302013_302014_302015_302016")
+   .put("ASA-6-302016", "CISCOFW302013_302014_302015_302016")
+   .put("ASA-6-302020", "CISCOFW302020_302021")
+   .put("ASA-6-302021", "CISCOFW302020_302021")
+   .put("ASA-6-305011", "CISCOFW305011")
+   .put("ASA-3-313001", "CISCOFW313001_313004_313008")
+   .put("ASA-3-313004", "CISCOFW313001_313004_313008")
+   .put("ASA-3-313008", "CISCOFW313001_313004_313008")
+   .put("ASA-4-313005", "CISCOFW313005")
+   .put("ASA-4-402117", "CISCOFW402117")
+   .put("ASA-4-402119", "CISCOFW402119")
+   .put("ASA-4-419001", "CISCOFW419001")
+   .put("ASA-4-419002", "CISCOFW419002")
+   .put("ASA-4-54", "CISCOFW54")
+   .put("ASA-6-602303", "CISCOFW602303_602304")
+   .put("ASA-6-602304", "CISCOFW602303_602304")
+   .put("ASA-7-710001", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-7-710002", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-7-710003", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-7-710005", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-7-710006", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-6-713172", "CISCOFW713172")
+   .put("ASA-4-733100", "CISCOFW733100")
+   .put("ASA-6-305012", "CISCOFW305012")
+   .put("ASA-7-609001", "CISCOFW609001")
+   .put("ASA-7-609002", "CISCOFW609002")
+.put("ASA-5-713041", "CISCOFW713041")
+.build();
+
+@Override
+

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

2016-10-01 Thread kylerichardson

Github user kylerichardson commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/276#discussion_r81454486
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/BasicAsaParser.java
 ---
@@ -0,0 +1,165 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.asa;
+
+import com.google.common.collect.ImmutableMap;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.commons.validator.routines.InetAddressValidator;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.utils.FieldValidators;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.*;
+import java.util.*;
+
+public class BasicAsaParser extends BasicParser {
+
+protected static final Logger LOG = 
LoggerFactory.getLogger(BasicAsaParser.class);
+
+private Grok asaGrok;
+
+private static final InetAddressValidator ipValidator = 
InetAddressValidator.getInstance();
+
+private static final Map patternMap = 
ImmutableMap.builder()
+.put("ASA-2-106001", "CISCOFW106001")
+   .put("ASA-2-106006", "CISCOFW106006_106007_106010")
+   .put("ASA-2-106007", "CISCOFW106006_106007_106010")
+   .put("ASA-2-106010", "CISCOFW106006_106007_106010")
+   .put("ASA-3-106014", "CISCOFW106014")
+   .put("ASA-6-106015", "CISCOFW106015")
+   .put("ASA-1-106021", "CISCOFW106021")
+   .put("ASA-4-106023", "CISCOFW106023")
+   .put("ASA-5-106100", "CISCOFW106100")
+   .put("ASA-6-110002", "CISCOFW110002")
+   .put("ASA-6-302010", "CISCOFW302010")
+   .put("ASA-6-302013", "CISCOFW302013_302014_302015_302016")
+   .put("ASA-6-302014", "CISCOFW302013_302014_302015_302016")
+   .put("ASA-6-302015", "CISCOFW302013_302014_302015_302016")
+   .put("ASA-6-302016", "CISCOFW302013_302014_302015_302016")
+   .put("ASA-6-302020", "CISCOFW302020_302021")
+   .put("ASA-6-302021", "CISCOFW302020_302021")
+   .put("ASA-6-305011", "CISCOFW305011")
+   .put("ASA-3-313001", "CISCOFW313001_313004_313008")
+   .put("ASA-3-313004", "CISCOFW313001_313004_313008")
+   .put("ASA-3-313008", "CISCOFW313001_313004_313008")
+   .put("ASA-4-313005", "CISCOFW313005")
+   .put("ASA-4-402117", "CISCOFW402117")
+   .put("ASA-4-402119", "CISCOFW402119")
+   .put("ASA-4-419001", "CISCOFW419001")
+   .put("ASA-4-419002", "CISCOFW419002")
+   .put("ASA-4-54", "CISCOFW54")
+   .put("ASA-6-602303", "CISCOFW602303_602304")
+   .put("ASA-6-602304", "CISCOFW602303_602304")
+   .put("ASA-7-710001", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-7-710002", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-7-710003", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-7-710005", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-7-710006", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-6-713172", "CISCOFW713172")
+   .put("ASA-4-733100", "CISCOFW733100")
+   .put("ASA-6-305012", "CISCOFW305012")
+   .put("ASA-7-609001", "CISCOFW609001")
+   .put("ASA-7-609002", "CISCOFW609002")
+.put("ASA-5-713041", "CISCOFW713041")
+.build();
+
+@Override
+

Re: [DISCUSS] Dockerize Metron

2016-10-01 Thread Casey Stella

I agree with that concern. Being able to debug a running topology has
helped in so many circumstances. Not sure how to accomplish this in a
dockerized environment.

On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic 
wrote:

> Any ideas on how debugging works when leveraging Docker? In spite of the
> classpath troubles, one of the benefits of the current single-JVM approach
> is that you can easily debug, set break points, etc within an IDE. Is that
> still doable? Seems like there might need to be some remote debugging magic
> done to accomplish this.
>
> On Fri, Sep 30, 2016 at 6:39 PM, James Sirota  wrote:
>
> > I agree with making an effort to create containers.  I suggest doing it
> on
> > a feature branch until we are feature complete and are able to migrate
> our
> > integration tests into a dockerized environment.
> >
> > 30.09.2016, 14:52, "Nick Allen" :
> > >  Relieve dependency version conflict issues? Umm, yes please. I'll
> take a
> > >  second helping of that too.
> > >
> > >  On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman 
> > wrote:
> > >
> > >>   I would like to open up a discussion around creating Docker images
> for
> > >>   Metron. Having this available would provide a leaner alternative to
> > the
> > >>   ansible/vagrant environment for development tesing (and even demoing
> > or
> > >>   exploring features). It could also relieve some of the dependency
> > version
> > >>   conflict issues that we've been experiencing when running
> integration
> > tests
> > >>   in a single JVM.
> > >>
> > >>   I would suggest the initial version be intended only for development
> > and
> > >>   testing purposes. The general approach could be to create an image
> for
> > >>   each service we depend on and use something like Docker compose to
> > package
> > >>   them together. A Dockerfile would either install the service from
> > scratch
> > >>   or extend a community image then add any Metron related dependencies
> > or
> > >>   configurations on top. The metron-deployment project code could be
> > used as
> > >>   a guide.
> > >>
> > >>   I would like to see these images added initially to support
> > development and
> > >>   testing:
> > >>
> > >>  - Kafka with topics preconfigured
> > >>  - Storm with metron topology assets installed
> > >>  - Zookeeper with paths created and sample configs loaded
> > >>  - HBase with sample enrichments and threat intel loaded
> > >>  - Elasticsearch configured for Metron
> > >>  - MySQL with databases/tables/users created and geo data loaded
> > >>
> > >>   Other images that could also be useful:
> > >>
> > >>  - Images for each sensor
> > >>  - Ambari?
> > >>  - Solr
> > >>
> > >>   Looking forward to hearing what everyone thinks.
> > >>
> > >>   Ryan Merriman
> > >
> > >  --
> > >  Nick Allen 
> >
> >
> > ---
> > Thank you,
> >
> > James Sirota
> > PPMC- Apache Metron (Incubating)
> > jsirota AT apache DOT org
> >
> > 30.09.2016, 14:52, "Nick Allen" :
> > > Relieve dependency version conflict issues? Umm, yes please. I'll take
> a
> > > second helping of that too.
> > >
> > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman 
> > wrote:
> > >
> > >>  I would like to open up a discussion around creating Docker images
> for
> > >>  Metron. Having this available would provide a leaner alternative to
> the
> > >>  ansible/vagrant environment for development tesing (and even demoing
> or
> > >>  exploring features). It could also relieve some of the dependency
> > version
> > >>  conflict issues that we've been experiencing when running integration
> > tests
> > >>  in a single JVM.
> > >>
> > >>  I would suggest the initial version be intended only for development
> > and
> > >>  testing purposes. The general approach could be to create an image
> for
> > >>  each service we depend on and use something like Docker compose to
> > package
> > >>  them together. A Dockerfile would either install the service from
> > scratch
> > >>  or extend a community image then add any Metron related dependencies
> or
> > >>  configurations on top. The metron-deployment project code could be
> > used as
> > >>  a guide.
> > >>
> > >>  I would like to see these images added initially to support
> > development and
> > >>  testing:
> > >>
> > >> - Kafka with topics preconfigured
> > >> - Storm with metron topology assets installed
> > >> - Zookeeper with paths created and sample configs loaded
> > >> - HBase with sample enrichments and threat intel loaded
> > >> - Elasticsearch configured for Metron
> > >> - MySQL with databases/tables/users created and geo data loaded
> > >>
> > >>  Other images that could also be useful:
> > >>
> > >> - Images for each sensor
> > >> - Ambari?
> > >> - Solr
> > >>
> > >>  Looking forward to hearing what everyone

Re: [DISCUSS] Dockerize Metron

2016-10-01 Thread Michael Miklavcic

Any ideas on how debugging works when leveraging Docker? In spite of the
classpath troubles, one of the benefits of the current single-JVM approach
is that you can easily debug, set break points, etc within an IDE. Is that
still doable? Seems like there might need to be some remote debugging magic
done to accomplish this.

On Fri, Sep 30, 2016 at 6:39 PM, James Sirota  wrote:

> I agree with making an effort to create containers.  I suggest doing it on
> a feature branch until we are feature complete and are able to migrate our
> integration tests into a dockerized environment.
>
> 30.09.2016, 14:52, "Nick Allen" :
> >  Relieve dependency version conflict issues? Umm, yes please. I'll take a
> >  second helping of that too.
> >
> >  On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman 
> wrote:
> >
> >>   I would like to open up a discussion around creating Docker images for
> >>   Metron. Having this available would provide a leaner alternative to
> the
> >>   ansible/vagrant environment for development tesing (and even demoing
> or
> >>   exploring features). It could also relieve some of the dependency
> version
> >>   conflict issues that we've been experiencing when running integration
> tests
> >>   in a single JVM.
> >>
> >>   I would suggest the initial version be intended only for development
> and
> >>   testing purposes. The general approach could be to create an image for
> >>   each service we depend on and use something like Docker compose to
> package
> >>   them together. A Dockerfile would either install the service from
> scratch
> >>   or extend a community image then add any Metron related dependencies
> or
> >>   configurations on top. The metron-deployment project code could be
> used as
> >>   a guide.
> >>
> >>   I would like to see these images added initially to support
> development and
> >>   testing:
> >>
> >>  - Kafka with topics preconfigured
> >>  - Storm with metron topology assets installed
> >>  - Zookeeper with paths created and sample configs loaded
> >>  - HBase with sample enrichments and threat intel loaded
> >>  - Elasticsearch configured for Metron
> >>  - MySQL with databases/tables/users created and geo data loaded
> >>
> >>   Other images that could also be useful:
> >>
> >>  - Images for each sensor
> >>  - Ambari?
> >>  - Solr
> >>
> >>   Looking forward to hearing what everyone thinks.
> >>
> >>   Ryan Merriman
> >
> >  --
> >  Nick Allen 
>
>
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
> 30.09.2016, 14:52, "Nick Allen" :
> > Relieve dependency version conflict issues? Umm, yes please. I'll take a
> > second helping of that too.
> >
> > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman 
> wrote:
> >
> >>  I would like to open up a discussion around creating Docker images for
> >>  Metron. Having this available would provide a leaner alternative to the
> >>  ansible/vagrant environment for development tesing (and even demoing or
> >>  exploring features). It could also relieve some of the dependency
> version
> >>  conflict issues that we've been experiencing when running integration
> tests
> >>  in a single JVM.
> >>
> >>  I would suggest the initial version be intended only for development
> and
> >>  testing purposes. The general approach could be to create an image for
> >>  each service we depend on and use something like Docker compose to
> package
> >>  them together. A Dockerfile would either install the service from
> scratch
> >>  or extend a community image then add any Metron related dependencies or
> >>  configurations on top. The metron-deployment project code could be
> used as
> >>  a guide.
> >>
> >>  I would like to see these images added initially to support
> development and
> >>  testing:
> >>
> >> - Kafka with topics preconfigured
> >> - Storm with metron topology assets installed
> >> - Zookeeper with paths created and sample configs loaded
> >> - HBase with sample enrichments and threat intel loaded
> >> - Elasticsearch configured for Metron
> >> - MySQL with databases/tables/users created and geo data loaded
> >>
> >>  Other images that could also be useful:
> >>
> >> - Images for each sensor
> >> - Ambari?
> >> - Solr
> >>
> >>  Looking forward to hearing what everyone thinks.
> >>
> >>  Ryan Merriman
> >
> > --
> > Nick Allen 
>
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

2016-10-01 Thread kylerichardson

Github user kylerichardson commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/276#discussion_r81453779
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/BasicAsaParser.java
 ---
@@ -0,0 +1,165 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers.asa;
+
+import com.google.common.collect.ImmutableMap;
+import oi.thekraken.grok.api.Grok;
+import oi.thekraken.grok.api.Match;
+import oi.thekraken.grok.api.exception.GrokException;
+import org.apache.commons.validator.routines.InetAddressValidator;
+import org.apache.metron.common.Constants;
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.utils.FieldValidators;
+import org.apache.metron.parsers.utils.SyslogUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.*;
+import java.util.*;
+
+public class BasicAsaParser extends BasicParser {
+
+protected static final Logger LOG = 
LoggerFactory.getLogger(BasicAsaParser.class);
+
+private Grok asaGrok;
+
+private static final InetAddressValidator ipValidator = 
InetAddressValidator.getInstance();
+
+private static final Map patternMap = 
ImmutableMap.builder()
+.put("ASA-2-106001", "CISCOFW106001")
+   .put("ASA-2-106006", "CISCOFW106006_106007_106010")
+   .put("ASA-2-106007", "CISCOFW106006_106007_106010")
+   .put("ASA-2-106010", "CISCOFW106006_106007_106010")
+   .put("ASA-3-106014", "CISCOFW106014")
+   .put("ASA-6-106015", "CISCOFW106015")
+   .put("ASA-1-106021", "CISCOFW106021")
+   .put("ASA-4-106023", "CISCOFW106023")
+   .put("ASA-5-106100", "CISCOFW106100")
+   .put("ASA-6-110002", "CISCOFW110002")
+   .put("ASA-6-302010", "CISCOFW302010")
+   .put("ASA-6-302013", "CISCOFW302013_302014_302015_302016")
+   .put("ASA-6-302014", "CISCOFW302013_302014_302015_302016")
+   .put("ASA-6-302015", "CISCOFW302013_302014_302015_302016")
+   .put("ASA-6-302016", "CISCOFW302013_302014_302015_302016")
+   .put("ASA-6-302020", "CISCOFW302020_302021")
+   .put("ASA-6-302021", "CISCOFW302020_302021")
+   .put("ASA-6-305011", "CISCOFW305011")
+   .put("ASA-3-313001", "CISCOFW313001_313004_313008")
+   .put("ASA-3-313004", "CISCOFW313001_313004_313008")
+   .put("ASA-3-313008", "CISCOFW313001_313004_313008")
+   .put("ASA-4-313005", "CISCOFW313005")
+   .put("ASA-4-402117", "CISCOFW402117")
+   .put("ASA-4-402119", "CISCOFW402119")
+   .put("ASA-4-419001", "CISCOFW419001")
+   .put("ASA-4-419002", "CISCOFW419002")
+   .put("ASA-4-54", "CISCOFW54")
+   .put("ASA-6-602303", "CISCOFW602303_602304")
+   .put("ASA-6-602304", "CISCOFW602303_602304")
+   .put("ASA-7-710001", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-7-710002", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-7-710003", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-7-710005", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-7-710006", 
"CISCOFW710001_710002_710003_710005_710006")
+   .put("ASA-6-713172", "CISCOFW713172")
+   .put("ASA-4-733100", "CISCOFW733100")
+   .put("ASA-6-305012", "CISCOFW305012")
+   .put("ASA-7-609001", "CISCOFW609001")
+   .put("ASA-7-609002", "CISCOFW609002")
+.put("ASA-5-713041", "CISCOFW713041")
+.build();
+
+@Override
+

[GitHub] incubator-metron issue #287: METRON-478: Add Michael Miklavcic, Justin Leet,...

2016-10-01 Thread mmiklavc

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/287
  
Confirmation - apache/asf-site branch has been updated with changes - 
http://metron.incubator.apache.org/community/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

2016-10-01 Thread kylerichardson

Github user kylerichardson commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/276#discussion_r81453224
  
--- Diff: 
metron-platform/metron-integration-test/src/main/sample/data/asa/raw/asa_raw ---
@@ -0,0 +1,128 @@
+<167>Jan  5 08:52:35 10.22.8.216 %ASA-7-609001: Built local-host 
inside:10.22.8.205
+<166>Jan  5 08:52:35 10.22.8.216 %ASA-6-302021: Teardown ICMP connection 
for faddr 10.22.8.74/0(LOCAL\user.name) gaddr 10.22.8.205/0 laddr 10.22.8.205/0
+<167>Jan  5 08:52:35 10.22.8.216 %ASA-7-609002: Teardown local-host 
inside:10.22.8.205 duration 0:00:00
+<142>Jan  5 08:52:35 10.22.8.201 %ASA-6-302014: Teardown TCP connection 
488167725 for Outside_VPN:147.111.72.16/26436 to DMZ-Inside:10.22.8.53/443 
duration 0:00:00 bytes 9687 TCP FINs
+<166>Jan  5 08:52:35 10.22.8.216 %ASA-6-302014: Teardown TCP connection 
212805593 for outside:10.22.8.223/59614(LOCAL\user.name) to 
inside:10.22.8.78/8102 duration 0:00:07 bytes 3433 TCP FINs (user.name)
+<174>Jan  5 14:52:35 10.22.8.212 %ASA-6-302013: Built inbound TCP 
connection 76245503 for outside:10.22.8.233/54209 (10.22.8.233/54209) to 
inside:198.111.72.238/443 (198.111.72.238/443) (user.name)
+<166>Jan  5 08:52:35 10.22.8.216 %ASA-6-302013: Built inbound TCP 
connection 212806031 for outside:10.22.8.17/58633 
(10.22.8.17/58633)(LOCAL\user.name) to inside:10.22.8.12/389 (10.22.8.12/389) 
(user.name)
+<142>Jan  5 08:52:35 10.22.8.201 %ASA-6-302014: Teardown TCP connection 
488168292 for DMZ-Inside:10.22.8.51/51231 to Inside-Trunk:10.22.8.174/40004 
duration 0:00:00 bytes 2103 TCP FINs
--- End diff --

I took the existing test data found in 
.../sample/data/SampleInput/AsaOutput and added to it data from some of my test 
devices. The data I added has been scrubbed/anonymized.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #287: METRON-478: Add Michael Miklavcic, Justi...

2016-10-01 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/287


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Complete steps to add a new parser

Re: [DISCUSS] Dockerize Metron

Re: [DISCUSS] Dockerize Metron

Re: [DISCUSS] Dockerize Metron

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

Re: [DISCUSS] Dockerize Metron

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

Re: [DISCUSS] Dockerize Metron

Re: [DISCUSS] Dockerize Metron

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

[GitHub] incubator-metron issue #287: METRON-478: Add Michael Miklavcic, Justin Leet,...

[GitHub] incubator-metron pull request #276: METRON-363 Fix Cisco ASA Parser

[GitHub] incubator-metron pull request #287: METRON-478: Add Michael Miklavcic, Justi...

15 matches

Site Navigation

Mail list logo

Footer information