[
https://issues.apache.org/jira/browse/METRON-370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15630684#comment-15630684
]
Michael Miklavcic commented on METRON-370:
------------------------------------------
I looked over the code for enrichments. It looks like this scenario of multiple
records landing in indexing could only occur by sending the same record through
the system multiple times, from ingestion. We do not currently implement
retries on failed enrichments, rather the tuple will get passed along
unenriched, and this is by design. That being said, it does not look like
having duplicate records could occur in a production environment, given the
current architecture. We may want to talk about adding a feature for replaying
failed enrichments and enabling progressive updates to existing indexes.
> Inconsistent Enrichment seen for the same Bro log event
> -------------------------------------------------------
>
> Key: METRON-370
> URL: https://issues.apache.org/jira/browse/METRON-370
> Project: Metron
> Issue Type: Bug
> Affects Versions: 0.2.2BETA
> Reporter: Neha Sinha
> Attachments: Screen Shot 2016-08-09 at 11.45.23 AM (2).png,
> enrichment-18-1470051231-worker-6700.log
>
>
> Hi,
> I am injecting the following Bro event and I see there is a diff in the
> enrichment keys getting generated .
> Bro Log :-
> =========================================
> HTTP | id.orig_p:49210 status_code:200 method:GET request_body_len:0
> id.resp_p:80 uri:/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764
> tags:[] uid:CRDObQRKAmoHCQq1a
> referrer:http://7oqnsnzwwnm6zb7y.gigapaysun.com/11iQmfg
> resp_mime_types:["image\/png"] trans_depth:3
> host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158
> response_body_len:1823 user_agent:Mozilla/4.0 (compatible; MSIE 8.0; Windows
> NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729;
> .NET CLR 3.0.30729; Media Center PC 6.0) ts:1.469686182899762E9
> id.resp_h:13.163.121.204 resp_fuids:["FQbZZfax5TLVX6M42"]
> Bro Index with Incomplete enrichment keys(Missing enrichment_ip_dst_addr keys)
> "bro_timestamp": "1.469686182899762E9",
> "status_code": 200,
> "ip_dst_port": 80,
> "threatinteljoinbolt:joiner:ts": "1470716948354",
> "enrichments:geo:ip_src_addr:longitude": "-97.822",
> "enrichmentsplitterbolt:splitter:begin:ts": "1470716948353",
> "enrichmentjoinbolt:joiner:ts": "1470716948353",
> "adapter:geoadapter:begin:ts": "1470716948353",
> "uid": "CRDObQRKAmoHCQq1a",
> "resp_mime_types": [
> "image/png"
> ],
> "trans_depth": 3,
> "protocol": "http",
> "source:type": "bro",
> "adapter:threatinteladapter:end:ts": "1470716948353",
> "original_string": "HTTP | id.orig_p:49210 status_code:200 method:GET
> request_body_len:0 id.resp_p:80
> uri:/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764 tags:[]
> uid:CRDObQRKAmoHCQq1a referrer:http://7oqnsnzwwnm6zb7y.gigapaysun.com/11iQmfg
> resp_mime_types:[\"image\\/png\"] trans_depth:3
> host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158
> response_body_len:1823 user_agent:Mozilla/4.0 (compatible; MSIE 8.0; Windows
> NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729;
> .NET CLR 3.0.30729; Media Center PC 6.0) ts:1.469686182899762E9
> id.resp_h:13.163.121.204 resp_fuids:[\"FQbZZfax5TLVX6M42\"]",
> "ip_dst_addr": "13.163.121.204",
> "enrichments:geo:ip_src_addr:locID": "223",
> "adapter:hostfromjsonlistadapter:end:ts": "1470716948353",
> "host": "7oqnsnzwwnm6zb7y.gigapaysun.com",
> "adapter:geoadapter:end:ts": "1470716948353",
> "enrichments:geo:ip_src_addr:latitude": "37.751",
> "ip_src_addr": "12.172.138.158",
> "threatintelsplitterbolt:splitter:end:ts": "1470716948353",
> "user_agent": "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1;
> WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR
> 3.0.30729; Media Center PC 6.0)",
> "resp_fuids": [
> "FQbZZfax5TLVX6M42"
> ],
> "timestamp": 1469686182899,
> "enrichments:geo:ip_src_addr:location_point": "37.751,-97.822",
> "method": "GET",
> "enrichmentsplitterbolt:splitter:end:ts": "1470716948353",
> "request_body_len": 0,
> "adapter:hostfromjsonlistadapter:begin:ts": "1470716948353",
> "enrichments:geo:ip_src_addr:country": "US",
> "uri": "/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764",
> "tags": [],
> "referrer": "http://7oqnsnzwwnm6zb7y.gigapaysun.com/11iQmfg";,
> "ip_src_port": 49210,
> "threatintelsplitterbolt:splitter:begin:ts": "1470716948353",
> "adapter:threatinteladapter:begin:ts": "1470716948353",
> "status_msg": "OK",
> "response_body_len": 1823
> Bro Index with Complete enrichment keys
> {
> "enrichments:geo:ip_dst_addr:locID": "223",
> "bro_timestamp": "1.469686182899762E9",
> "status_code": 200,
> "enrichments:geo:ip_dst_addr:location_point": "37.751,-97.822",
> "ip_dst_port": 80,
> "threatinteljoinbolt:joiner:ts": "1470717673183",
> "enrichments:geo:ip_src_addr:longitude": "-97.822",
> "enrichmentsplitterbolt:splitter:begin:ts": "1470717673183",
> "enrichmentjoinbolt:joiner:ts": "1470717673183",
> "adapter:geoadapter:begin:ts": "1470717673183",
> "enrichments:geo:ip_dst_addr:latitude": "37.751",
> "uid": "CRDObQRKAmoHCQq1a",
> "resp_mime_types": [
> "image/png"
> ],
> "trans_depth": 3,
> "protocol": "http",
> "source:type": "bro",
> "adapter:threatinteladapter:end:ts": "1470717673183",
> "original_string": "HTTP | id.orig_p:49210 status_code:200 method:GET
> request_body_len:0 id.resp_p:80
> uri:/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764 tags:[]
> uid:CRDObQRKAmoHCQq1a referrer:http://7oqnsnzwwnm6zb7y.gigapaysun.com/11iQmfg
> resp_mime_types:[\"image\\/png\"] trans_depth:3
> host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158
> response_body_len:1823 user_agent:Mozilla/4.0 (compatible; MSIE 8.0; Windows
> NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729;
> .NET CLR 3.0.30729; Media Center PC 6.0) ts:1.469686182899762E9
> id.resp_h:13.163.121.204 resp_fuids:[\"FQbZZfax5TLVX6M42\"]",
> "ip_dst_addr": "13.163.121.204",
> "enrichments:geo:ip_src_addr:locID": "223",
> "adapter:hostfromjsonlistadapter:end:ts": "1470717673183",
> "host": "7oqnsnzwwnm6zb7y.gigapaysun.com",
> "adapter:geoadapter:end:ts": "1470717673183",
> "enrichments:geo:ip_src_addr:latitude": "37.751",
> "ip_src_addr": "12.172.138.158",
> "threatintelsplitterbolt:splitter:end:ts": "1470717673183",
> "enrichments:geo:ip_dst_addr:longitude": "-97.822",
> "user_agent": "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1;
> WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR
> 3.0.30729; Media Center PC 6.0)",
> "resp_fuids": [
> "FQbZZfax5TLVX6M42"
> ],
> "timestamp": 1469686182899,
> "enrichments:geo:ip_src_addr:location_point": "37.751,-97.822",
> "method": "GET",
> "enrichmentsplitterbolt:splitter:end:ts": "1470717673183",
> "request_body_len": 0,
> "adapter:hostfromjsonlistadapter:begin:ts": "1470717673183",
> "enrichments:geo:ip_src_addr:country": "US",
> "uri": "/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764",
> "tags": [],
> "referrer": "http://7oqnsnzwwnm6zb7y.gigapaysun.com/11iQmfg";,
> "ip_src_port": 49210,
> "threatintelsplitterbolt:splitter:begin:ts": "1470717673183",
> "adapter:threatinteladapter:begin:ts": "1470717673183",
> "status_msg": "OK",
> "enrichments:geo:ip_dst_addr:country": "US",
> "response_body_len": 1823
> }
> The JSON diff shows that the enrichment for ip_dst_addr is missing for the
> first index that gets created for the same bro event.(However enrichment
> happened all good for ip_src_addr)
> I have attached the enrichment logs enrichment-18-1470051231-worker-6700.log
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
