Mohan created METRON-1466:
-----------------------------
Summary: The Elastic Search index are not getting generated in the
latest metron deployments for all the topologies
Key: METRON-1466
URL: https://issues.apache.org/jira/browse/METRON-1466
Project: Metron
Issue Type: Bug
Environment: Metron secured cluster
Reporter: Mohan
Attachments: bro_worker.log, enrichmnts_worker.log, indexing_worker.log
The Elastic Search index are not getting generated in the latest metron
deployments for all the topologies ,
I see that the data is getting published to the respective topics and also
getting enriched .
below are the console consumer output for topics "bro" and "enrichments" I dont
get anything for "indexing" topic .
{code:java}
./kafka-console-consumer.sh --zookeeper
ctr-e138-1518143905142-61778-01-000002.hwx.site:2181 --security-protocol
PLAINTEXTSASL --topic bro
{"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC
6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
{"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC
6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
{"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC
6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
{"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC
6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
{"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC
6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
./kafka-console-consumer.sh --zookeeper
ctr-e138-1518143905142-61778-01-000002.hwx.site:2181 --security-protocol
PLAINTEXTSASL --topic enrichments
{"bro_timestamp":"1469686182.899762","status_code":200,"method":"GET","ip_dst_port":80,"request_body_len":0,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"source.type":"bro","uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"protocol":"http","original_string":"HTTP
| id.orig_p:49210 status_code:200 method:GET request_body_len:0 id.resp_p:80
uri:\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764 tags:[]
uid:CRDObQRKAmoHCQq1a
referrer:http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg
resp_mime_types:[\"image\\\/png\"] trans_depth:3
host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158
response_body_len:1823 user_agent:Mozilla\/4.0 (compatible; MSIE 8.0; Windows
NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729;
.NET CLR 3.0.30729; Media Center PC 6.0) ts:1469686182.899762
id.resp_h:13.163.121.204
resp_fuids:[\"FQbZZfax5TLVX6M42\"]","ip_dst_addr":"13.163.121.204","ip_src_port":49210,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","guid":"897e534c-ed6b-458c-9b2d-8749a68ed447","response_body_len":1823,"ip_src_addr":"12.172.138.158","user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC
6.0)","resp_fuids":["FQbZZfax5TLVX6M42"],"timestamp":1469686182899}
{"bro_timestamp":"1469686182.899762","status_code":200,"method":"GET","ip_dst_port":80,"request_body_len":0,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"source.type":"bro","uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"protocol":"http","original_string":"HTTP
| id.orig_p:49210 status_code:200 method:GET request_body_len:0 id.resp_p:80
uri:\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764 tags:[]
uid:CRDObQRKAmoHCQq1a
referrer:http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg
resp_mime_types:[\"image\\\/png\"] trans_depth:3
host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158
response_body_len:1823 user_agent:Mozilla\/4.0 (compatible; MSIE 8.0; Windows
NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729;
.NET CLR 3.0.30729; Media Center PC 6.0) ts:1469686182.899762
id.resp_h:13.163.121.204
resp_fuids:[\"FQbZZfax5TLVX6M42\"]","ip_dst_addr":"13.163.121.204","ip_src_port":49210,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","guid":"6caaab54-e78d-4d31-9b2c-5c2bb25d609f","response_body_len":1823,"ip_src_addr":"12.172.138.158","user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC
6.0)","resp_fuids":["FQbZZfax5TLVX6M42"],"timestamp":1469686182899}
{"bro_timestamp":"1469686182.899762","status_code":200,"method":"GET","ip_dst_port":80,"request_body_len":0,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"source.type":"bro","uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"protocol":"http","original_string":"HTTP
| id.orig_p:49210 status_code:200 method:GET request_body_len:0 id.resp_p:80
uri:\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764 tags:[]
uid:CRDObQRKAmoHCQq1a
referrer:http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg
resp_mime_types:[\"image\\\/png\"] trans_depth:3
host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158
response_body_len:1823 user_agent:Mozilla\/4.0 (compatible; MSIE 8.0; Windows
NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729;
.NET CLR 3.0.30729; Media Center PC 6.0) ts:1469686182.899762
id.resp_h:13.163.121.204
resp_fuids:[\"FQbZZfax5TLVX6M42\"]","ip_dst_addr":"13.163.121.204","ip_src_port":49210,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","guid":"a7475614-789a-4221-88cb-0734ef5fdc0a","response_body_len":1823,"ip_src_addr":"12.172.138.158","user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC
6.0)","resp_fuids":["FQbZZfax5TLVX6M42"],"timestamp":1469686182899}
{"bro_timestamp":"1469686182.899762","status_code":200,"method":"GET","ip_dst_port":80,"request_body_len":0,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"source.type":"bro","uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"protocol":"http","original_string":"HTTP
| id.orig_p:49210 status_code:200 method:GET request_body_len:0 id.resp_p:80
uri:\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764 tags:[]
uid:CRDObQRKAmoHCQq1a
referrer:http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg
resp_mime_types:[\"image\\\/png\"] trans_depth:3
host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158
response_body_len:1823 user_agent:Mozilla\/4.0 (compatible; MSIE 8.0; Windows
NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729;
.NET CLR 3.0.30729; Media Center PC 6.0) ts:1469686182.899762
id.resp_h:13.163.121.204
resp_fuids:[\"FQbZZfax5TLVX6M42\"]","ip_dst_addr":"13.163.121.204","ip_src_port":49210,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","guid":"f05fa5de-5549-4293-85cf-006c66dba5d4","response_body_len":1823,"ip_src_addr":"12.172.138.158","user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC
6.0)","resp_fuids":["FQbZZfax5TLVX6M42"],"timestamp":1469686182899}
{"bro_timestamp":"1469686182.899762","status_code":200,"method":"GET","ip_dst_port":80,"request_body_len":0,"uri":"\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764","tags":[],"source.type":"bro","uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"protocol":"http","original_string":"HTTP
| id.orig_p:49210 status_code:200 method:GET request_body_len:0 id.resp_p:80
uri:\/picture.php?k=11iqmfg&b7f2a994c3eaaf014608b272c46cf764 tags:[]
uid:CRDObQRKAmoHCQq1a
referrer:http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg
resp_mime_types:[\"image\\\/png\"] trans_depth:3
host:7oqnsnzwwnm6zb7y.gigapaysun.com status_msg:OK id.orig_h:12.172.138.158
response_body_len:1823 user_agent:Mozilla\/4.0 (compatible; MSIE 8.0; Windows
NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729;
.NET CLR 3.0.30729; Media Center PC 6.0) ts:1469686182.899762
id.resp_h:13.163.121.204
resp_fuids:[\"FQbZZfax5TLVX6M42\"]","ip_dst_addr":"13.163.121.204","ip_src_port":49210,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","guid":"116cb86a-2f41-48a8-8cc0-ee15d25d8db6","response_body_len":1823,"ip_src_addr":"12.172.138.158","user_agent":"Mozilla\/4.0
(compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR
2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC
6.0)","resp_fuids":["FQbZZfax5TLVX6M42"],"timestamp":1469686182899}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
