I have attached the output of this dump
/usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP
On Thu, Nov 9, 2017 at 12:06 AM, zeo...@gmail.com <zeo...@gmail.com> wrote:
> What is the output of:
>
> /usr/metron/0.4.1/bin/zk_load_configs.sh -z node1:2181 -m DUMP
>
> ?
>
> Jon
>
> On Wed, Nov 8, 2017 at 1:49 PM Syed Hammad Tahir <mscs16...@itu.edu.pk>
> wrote:
>
>> This is the script/command i used
>>
>> sudo cat snort.out |
>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh
>> --broker-list node1:6667 --topic snort
>>
>> On Wed, Nov 8, 2017 at 11:18 PM, Syed Hammad Tahir <mscs16...@itu.edu.pk>
>> wrote:
>>
>>> sudo cat snort.out |
>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh
>>> --broker-list node1:6667 --topic snort
>>>
>>> On Wed, Nov 8, 2017 at 11:14 PM, Otto Fowler <ottobackwa...@gmail.com>
>>> wrote:
>>>
>>>> What topic? what are the parameters you are calling the script with?
>>>>
>>>>
>>>>
>>>> On November 8, 2017 at 13:12:56, Syed Hammad Tahir (
>>>> mscs16...@itu.edu.pk) wrote:
>>>>
>>>> The metron installation I have (single node based vm install) comes
>>>> with sensor stubs. I assume that everything has already been done for those
>>>> stub sensors to push the canned data. I am doing the similar thing,
>>>> directly pushing the preformatted canned data to kafka topic. I can see the
>>>> logs in kibana dashboard when I start stub sensor from monit but then I
>>>> push the same logs myself, those errors pop that I have shown earlier.
>>>>
>>>> On Wed, Nov 8, 2017 at 11:08 PM, Casey Stella <ceste...@gmail.com>
>>>> wrote:
>>>>
>>>>> How did you start the snort parser topology and what's the parser
>>>>> config (in zookeeper)?
>>>>>
>>>>> On Wed, Nov 8, 2017 at 1:06 PM, Syed Hammad Tahir <
>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>
>>>>>> This is what I am doing
>>>>>>
>>>>>> sudo cat snort.out |
>>>>>> /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh
>>>>>> --broker-list node1:6667 --topic snort
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 8, 2017 at 10:44 PM, Casey Stella <ceste...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Are you directly writing to the "indexing" kafka topic from the
>>>>>>> parser or from some other source? It looks like there are some records
>>>>>>> in
>>>>>>> kafka that are not JSON. By the time it gets to the indexing kafka
>>>>>>> topic,
>>>>>>> it should be a JSON map. The parser topology emits that JSON map and
>>>>>>> then
>>>>>>> the enrichments topology enrich that map and emits the enriched map to
>>>>>>> the
>>>>>>> indexing topic.
>>>>>>>
>>>>>>> On Wed, Nov 8, 2017 at 12:21 PM, Syed Hammad Tahir <
>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>
>>>>>>>> No I am no longer seeing the parsing topology error, here is the
>>>>>>>> full stack trace
>>>>>>>>
>>>>>>>> from hdfsindexingbolt in indexing topology
>>>>>>>>
>>>>>>>> [image: Inline image 1]
>>>>>>>>
>>>>>>>> from indexingbolt in indexing topology
>>>>>>>>
>>>>>>>> [image: Inline image 2]
>>>>>>>>
>>>>>>>> On Wed, Nov 8, 2017 at 10:08 PM, Otto Fowler <
>>>>>>>> ottobackwa...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> What Casey said. We need the whole stack trace.
>>>>>>>>> Also, are you saying that you are no longer seeing the parser
>>>>>>>>> topology error?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On November 8, 2017 at 11:39:06, Casey Stella (ceste...@gmail.com)
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> If you click on the port (6704) there in those errors, what's the
>>>>>>>>> full stacktrace (that starts with the suggestion you file a JIRA)?
>>>>>>>>>
>>>>>>>>> What this means is that an exception is bleeding from the
>>>>>>>>> individual writer into the writer component (It should be handled in
>>>>>>>>> the
>>>>>>>>> writer itself). The fact that it's happening for both HDFS and ES is
>>>>>>>>> telling as well and I'm very interested in the full stacktrace there
>>>>>>>>> because it'll have the wrapped exception from the individual writer
>>>>>>>>> included.
>>>>>>>>>
>>>>>>>>> On Wed, Nov 8, 2017 at 11:24 AM, Syed Hammad Tahir <
>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>
>>>>>>>>>> OK I did what Zeolla said, cat snort.out | kafka producer ....
>>>>>>>>>> and now the error at storm parser topology is gone but I am now
>>>>>>>>>> seeing this
>>>>>>>>>> at the indexing toology
>>>>>>>>>>
>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Nov 8, 2017 at 8:25 PM, Syed Hammad Tahir <
>>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>>
>>>>>>>>>>> this is a single line I am trying to push
>>>>>>>>>>> 01/11/17-20:49:18.107168 ,1,999158,0,"'snort test
>>>>>>>>>>> alert'",TCP,192.168.66.1,49581,192.168.66.121,22,0A:00:
>>>>>>>>>>> 27:00:00:00,08:00:27:E8:B0:7A,0x5A,***AP***,0x1E396BFC,
>>>>>>>>>>> 0x56900BB6,,0x1000,64,10,23403,76,77824,,,,
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Nov 8, 2017 at 5:30 PM, zeo...@gmail.com <
>>>>>>>>>>> zeo...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I would download the entire snort.out file and run cat
>>>>>>>>>>>> snort.out | kafka-console-producer.sh ... to make sure there are
>>>>>>>>>>>> no copy
>>>>>>>>>>>> paste problems
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Nov 8, 2017, 06:59 Otto Fowler <ottobackwa...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> The snort parser is coded to support dates in this format:
>>>>>>>>>>>>>
>>>>>>>>>>>>> private static String defaultDateFormat =
>>>>>>>>>>>>> "MM/dd/yy-HH:mm:ss.SSSSSS";
>>>>>>>>>>>>> private transient DateTimeFormatter dateTimeFormatter;
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> If your records are in dd/MM/yy- format, then you may see
>>>>>>>>>>>>> this error I believe.
>>>>>>>>>>>>> Can you verify the timestamp field’s format?
>>>>>>>>>>>>>
>>>>>>>>>>>>> If this is the case, then you will need to modify the default
>>>>>>>>>>>>> log timestamp format for snort in the short term.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On November 8, 2017 at 06:09:11, Otto Fowler (
>>>>>>>>>>>>> ottobackwa...@gmail.com) wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can you post what the value of the ‘timestamp’ field/column is
>>>>>>>>>>>>> for a piece of data that is failing
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On November 8, 2017 at 03:55:47, Syed Hammad Tahir (
>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now I am pretty sure that the issue is the format of the logs
>>>>>>>>>>>>> I am trying to push
>>>>>>>>>>>>>
>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can someone tell me the location of snort stub canned data
>>>>>>>>>>>>> file? Maybe I could see its formatting and try following the same
>>>>>>>>>>>>> thing.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 10:13 PM, Syed Hammad Tahir <
>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> thats how I am pushing my logs to kafka topic
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> After running this command, I copy paste a few lines from
>>>>>>>>>>>>>> here: https://raw.githubusercontent.com/apache/metron/master/
>>>>>>>>>>>>>> metron-deployment/roles/sensor-stubs/files/snort.out
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> like this
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am not getting any error here. I can also see these lines
>>>>>>>>>>>>>> pushed out via kafka consumer under topic of snort.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This was the mechanism I am using to push the logs.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 7:18 PM, Otto Fowler <
>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What I mean is this:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I *think* you have tried both messages coming from snort
>>>>>>>>>>>>>>> through some setup ( getting pushed to kafka ), which I think
>>>>>>>>>>>>>>> of as live.
>>>>>>>>>>>>>>> I also think you have manually pushed messages, where you see
>>>>>>>>>>>>>>> this error.
>>>>>>>>>>>>>>> So what I am asking is if you see the same errors for things
>>>>>>>>>>>>>>> that are automatically pushed to kafka as you do when you
>>>>>>>>>>>>>>> manual push them.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On November 7, 2017 at 08:51:41, Syed Hammad Tahir (
>>>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Yes, If the messages cannot be parsed then that would be a
>>>>>>>>>>>>>>> problem. If you see this error with your ‘live’ messages as
>>>>>>>>>>>>>>> well then that
>>>>>>>>>>>>>>> could be it.
>>>>>>>>>>>>>>> I wonder if the issue is with the date format?"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If by 'live' messages you mean the time I push them into
>>>>>>>>>>>>>>> kafka topic then no, I dont see any error at that time. If
>>>>>>>>>>>>>>> 'live' means
>>>>>>>>>>>>>>> something else here then please tell me what could it be.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 5:07 PM, Otto Fowler <
>>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes, If the messages cannot be parsed then that would be a
>>>>>>>>>>>>>>>> problem. If you see this error with your ‘live’ messages as
>>>>>>>>>>>>>>>> well then that
>>>>>>>>>>>>>>>> could be it.
>>>>>>>>>>>>>>>> I wonder if the issue is with the date format?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> You need to confirm that you see these same errors with the
>>>>>>>>>>>>>>>> live data or not.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Remember, the flow is like this
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> snort -> ??? -> Kafka -> Storm Parser Topology -> kafka ->
>>>>>>>>>>>>>>>> Storm Enrichment Topology -> Kafka -> Storm Indexing Topology
>>>>>>>>>>>>>>>> -> HDFS |
>>>>>>>>>>>>>>>> ElasticSearch
>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>> Kibana <-> Elastic Search
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Any point in this chain could fail and result in Kibana not
>>>>>>>>>>>>>>>> seeing things.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On November 7, 2017 at 01:57:19, Syed Hammad Tahir (
>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> could this be related to why I am unable to see logs in
>>>>>>>>>>>>>>>> kibana dashboard?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am copying a few lines from here
>>>>>>>>>>>>>>>> https://raw.githubusercontent.com/apache/metron/master/
>>>>>>>>>>>>>>>> metron-deployment/roles/sensor-stubs/files/snort.out
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> and then pushing them to snort kafka topic.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> THis is some error I am seeing in stormUI parser bolt in
>>>>>>>>>>>>>>>> snort section:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Nov 7, 2017 at 11:49 AM, Syed Hammad Tahir <
>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I guess I have hit a dead end. I am not able to get the
>>>>>>>>>>>>>>>>> snort logs in kibana dashboard. Any help will be appreciated.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Nov 6, 2017 at 1:24 PM, Syed Hammad Tahir <
>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I guess this (metron.log) in /var/log/elasticsearch/ is
>>>>>>>>>>>>>>>>>> also relevant
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Nov 6, 2017 at 11:46 AM, Syed Hammad Tahir <
>>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cluster health by index shows this:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> looks like some shard is unassigned and that is related
>>>>>>>>>>>>>>>>>>> to snort. Could it be the logs I was pushing to kafka topic
>>>>>>>>>>>>>>>>>>> earlier?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Nov 6, 2017 at 10:47 AM, Syed Hammad Tahir <
>>>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This is what I see here. What should I be looking at
>>>>>>>>>>>>>>>>>>>> here?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Nov 6, 2017 at 10:33 AM, Syed Hammad Tahir <
>>>>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> hi, I am back at work. lets see if i can find
>>>>>>>>>>>>>>>>>>>>> something in logs
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sat, Nov 4, 2017 at 6:38 PM, zeo...@gmail.com <
>>>>>>>>>>>>>>>>>>>>> zeo...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> It looks like your ES cluster has a health of Red, so
>>>>>>>>>>>>>>>>>>>>>> there's your problem. I would go look in
>>>>>>>>>>>>>>>>>>>>>> /var/log/elasticsearch/ at some
>>>>>>>>>>>>>>>>>>>>>> logs.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 3, 2017 at 12:19 PM Syed Hammad Tahir <
>>>>>>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> ---------- Forwarded message ----------
>>>>>>>>>>>>>>>>>>>>>>> From: Syed Hammad Tahir <mscs16...@itu.edu.pk>
>>>>>>>>>>>>>>>>>>>>>>> Date: Fri, Nov 3, 2017 at 5:07 PM
>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Snort Logs
>>>>>>>>>>>>>>>>>>>>>>> To: Otto Fowler <ottobackwa...@gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> NVM, I have installed the elastic search head. Now
>>>>>>>>>>>>>>>>>>>>>>> where do I go in this to find out why I cant see the
>>>>>>>>>>>>>>>>>>>>>>> snort logs in kibana
>>>>>>>>>>>>>>>>>>>>>>> dashboard, pushed to snort topic via kafka producer?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Fri, Nov 3, 2017 at 5:03 PM, Otto Fowler <
>>>>>>>>>>>>>>>>>>>>>>> ottobackwa...@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> You can install it into the chrome web browser from
>>>>>>>>>>>>>>>>>>>>>>>> the play store.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On November 3, 2017 at 07:47:47, Syed Hammad Tahir (
>>>>>>>>>>>>>>>>>>>>>>>> mscs16...@itu.edu.pk) wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> And how do I install elasticsearch head on the
>>>>>>>>>>>>>>>>>>>>>>>> vagrant VM?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> Jon
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> --
>
> Jon
>
GLOBAL Config: global
{
"es.clustername" : "metron",
"es.ip" : "node1:9300",
"es.date.format" : "yyyy.MM.dd.HH",
"parser.error.topic" : "indexing",
"update.hbase.table" : "metron_update",
"update.hbase.cf" : "t",
"profiler.client.period.duration" : "15",
"profiler.client.period.duration.units" : "MINUTES",
"geo.hdfs.file" : "/apps/metron/geo/default/GeoLite2-City.mmdb.gz"
}
PARSER Config: websphere
{
"parserClassName":"org.apache.metron.parsers.websphere.GrokWebSphereParser",
"sensorTopic":"websphere",
"parserConfig":
{
"grokPath":"/patterns/websphere",
"patternLabel":"WEBSPHERE",
"timestampField":"timestamp_string",
"dateFormat":"yyyy MMM dd HH:mm:ss"
}
}
PARSER Config: jsonMap
{
"parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
"sensorTopic":"jsonMap"
}
PARSER Config: squid
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "squid",
"parserConfig": {
"grokPath": "/patterns/squid",
"patternLabel": "SQUID_DELIMITED",
"timestampField": "timestamp"
},
"fieldTransformations" : [
{
"transformation" : "STELLAR"
,"output" : [ "full_hostname", "domain_without_subdomains" ]
,"config" : {
"full_hostname" : "URL_TO_HOST(url)"
,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)"
}
}
]
}
PARSER Config: snort
{
"parserClassName":"org.apache.metron.parsers.snort.BasicSnortParser",
"sensorTopic":"snort",
"parserConfig": {}
}
PARSER Config: asa
{
"parserClassName": "org.apache.metron.parsers.asa.BasicAsaParser",
"sensorTopic": "asa",
"parserConfig": {
"deviceTimeZone": "UTC"
}
}
PARSER Config: bro
{
"parserClassName":"org.apache.metron.parsers.bro.BasicBroParser",
"sensorTopic":"bro",
"parserConfig": {}
}
PARSER Config: yaf
{
"parserClassName":"org.apache.metron.parsers.GrokParser",
"sensorTopic":"yaf",
"fieldTransformations" : [
{
"input" : "protocol"
,"transformation": "IP_PROTOCOL"
}
],
"parserConfig":
{
"grokPath":"/patterns/yaf",
"patternLabel":"YAF_DELIMITED",
"timestampField":"start_time",
"timeFields": ["start_time", "end_time"],
"dateFormat":"yyyy-MM-dd HH:mm:ss.S"
}
}
INDEXING Config: websphere
{
"hdfs" : {
"index": "websphere",
"batchSize": 5,
"enabled" : true
},
"elasticsearch" : {
"index": "websphere",
"batchSize": 5,
"enabled" : true
},
"solr" : {
"index": "websphere",
"batchSize": 5,
"enabled" : true
}
}
INDEXING Config: error
{
"hdfs" : {
"index": "error",
"batchSize": 5,
"enabled" : true
},
"elasticsearch" : {
"index": "error",
"batchSize": 5,
"enabled" : true
},
"solr" : {
"index": "error",
"batchSize": 5,
"enabled" : true
}
}
INDEXING Config: snort
{
"hdfs" : {
"index": "snort",
"batchSize": 1,
"enabled" : true
},
"elasticsearch" : {
"index": "snort",
"batchSize": 1,
"enabled" : true
},
"solr" : {
"index": "snort",
"batchSize": 1,
"enabled" : true
}
}
INDEXING Config: asa
{
"hdfs" : {
"index": "asa",
"batchSize": 5,
"enabled" : true
},
"elasticsearch" : {
"index": "asa",
"batchSize": 5,
"enabled" : true
},
"solr" : {
"index": "asa",
"batchSize": 5,
"enabled" : true
}
}
INDEXING Config: bro
{
"hdfs" : {
"index": "bro",
"batchSize": 5,
"enabled" : true
},
"elasticsearch" : {
"index": "bro",
"batchSize": 5,
"enabled" : true
},
"solr" : {
"index": "bro",
"batchSize": 5,
"enabled" : true
}
}
INDEXING Config: yaf
{
"hdfs" : {
"index": "yaf",
"batchSize": 5,
"enabled" : true
},
"elasticsearch" : {
"index": "yaf",
"batchSize": 5,
"enabled" : true
},
"solr" : {
"index": "yaf",
"batchSize": 5,
"enabled" : true
}
}
ENRICHMENT Config: websphere
{
"enrichment": {
"fieldMap": {
"geo": [
"ip_src_addr"
],
"host": [
"ip_src_addr"
]
},
"fieldToTypeMap": {
"ip_src_addr": [
"playful_classification"
]
}
}
}
ENRICHMENT Config: snort
{
"enrichment" : {
"fieldMap":
{
"geo": ["ip_dst_addr", "ip_src_addr"],
"host": ["host"]
}
},
"threatIntel" : {
"fieldMap":
{
"hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
},
"fieldToTypeMap":
{
"ip_src_addr" : ["malicious_ip"],
"ip_dst_addr" : ["malicious_ip"]
},
"triageConfig" : {
"riskLevelRules" : [
{
"rule" : "not(IN_SUBNET(ip_dst_addr, '192.168.0.0/24'))",
"score" : 10
}
],
"aggregator" : "MAX"
}
}
}
ENRICHMENT Config: asa
{
"enrichment" : {
"fieldMap": {
"geo": ["ip_dst_addr", "ip_src_addr"]
}
}
}
ENRICHMENT Config: bro
{
"enrichment" : {
"fieldMap": {
"geo": ["ip_dst_addr", "ip_src_addr"],
"host": ["host"]
}
},
"threatIntel": {
"fieldMap": {
"hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
},
"fieldToTypeMap": {
"ip_src_addr" : ["malicious_ip"],
"ip_dst_addr" : ["malicious_ip"]
}
}
}
ENRICHMENT Config: yaf
{
"enrichment" : {
"fieldMap":
{
"geo": ["ip_dst_addr", "ip_src_addr"],
"host": ["host"]
}
},
"threatIntel": {
"fieldMap":
{
"hbaseThreatIntel": ["ip_src_addr", "ip_dst_addr"]
},
"fieldToTypeMap":
{
"ip_src_addr" : ["malicious_ip"],
"ip_dst_addr" : ["malicious_ip"]
}
}
}
