subject:"\[jira\] \[Commented\] \(METRON\-1466\) The Elastic Search index are not getting generated in the latest metron deployments for all the topologies"

[jira] [Commented] (METRON-1466) The Elastic Search index are not getting generated in the latest metron deployments for all the topologies

2018-03-02 Thread Michael Miklavcic (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383944#comment-16383944
 ] 

Michael Miklavcic commented on METRON-1466:
---

Looks like trouble with the maxmind DB

{code}
2018-03-02 17:36:56.357 o.a.m.e.a.g.GeoLiteDatabase 
Thread-39-threatIntelJoinBolt-executor[15 15] [ERROR] [Metron] Unable to open 
new database file /apps/metron/geo/default/GeoLite2-City.mmdb.gz
java.util.zip.ZipException: Not in GZIP format
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:165) 
~[?:1.8.0_161]
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:79) 
~[?:1.8.0_161]
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:91) 
~[?:1.8.0_161]
at 
org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase.update(GeoLiteDatabase.java:136)
 [stormjar.jar:?]
at 
org.apache.metron.enrichment.bolt.ThreatIntelJoinBolt.prepare(ThreatIntelJoinBolt.java:110)
 [stormjar.jar:?]
at 
org.apache.metron.enrichment.bolt.JoinBolt.prepare(JoinBolt.java:101) 
[stormjar.jar:?]
at 
org.apache.storm.daemon.executor$fn__10193$fn__10206.invoke(executor.clj:794) 
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:482) 
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2018-03-02 17:36:56.357 o.a.m.e.a.g.GeoLiteDatabase 
Thread-19-geoEnrichmentBolt-executor[6 6] [ERROR] [Metron] Unable to open new 
database file /apps/metron/geo/default/GeoLite2-City.mmdb.gz
java.util.zip.ZipException: Not in GZIP format
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:165) 
~[?:1.8.0_161]
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:79) 
~[?:1.8.0_161]
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:91) 
~[?:1.8.0_161]
at 
org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase.update(GeoLiteDatabase.java:136)
 [stormjar.jar:?]
at 
org.apache.metron.enrichment.adapters.geo.GeoAdapter.initializeAdapter(GeoAdapter.java:59)
 [stormjar.jar:?]
at 
org.apache.metron.enrichment.bolt.GenericEnrichmentBolt.prepare(GenericEnrichmentBolt.java:157)
 [stormjar.jar:?]
at 
org.apache.storm.daemon.executor$fn__10193$fn__10206.invoke(executor.clj:794) 
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:482) 
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2018-03-02 17:36:56.357 o.a.m.e.a.g.GeoLiteDatabase Curator-TreeCache-11 
[ERROR] [Metron] Unable to open new database file 
/apps/metron/geo/default/GeoLite2-City.mmdb.gz
java.util.zip.ZipException: Not in GZIP format
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:165) 
~[?:1.8.0_161]
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:79) 
~[?:1.8.0_161]
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:91) 
~[?:1.8.0_161]
at 
org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase.update(GeoLiteDatabase.java:136)
 ~[stormjar.jar:?]
at 
org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase.updateIfNecessary(GeoLiteDatabase.java:114)
 ~[stormjar.jar:?]
at 
org.apache.metron.enrichment.adapters.geo.GeoAdapter.updateAdapter(GeoAdapter.java:65)
 ~[stormjar.jar:?]
at 
org.apache.metron.enrichment.bolt.GenericEnrichmentBolt.reloadCallback(GenericEnrichmentBolt.java:133)
 ~[stormjar.jar:?]
at 
org.apache.metron.common.zookeeper.configurations.ConfigurationsUpdater.reloadCallback(ConfigurationsUpdater.java:145)
 ~[stormjar.jar:?]
at 
org.apache.metron.common.zookeeper.configurations.ConfigurationsUpdater.update(ConfigurationsUpdater.java:74)
 ~[stormjar.jar:?]
at 
org.apache.metron.zookeeper.SimpleEventListener.childEvent(SimpleEventListener.java:118)
 [stormjar.jar:?]
at 
org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:685)
 [stormjar.jar:?]
at 
org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:679)
 [stormjar.jar:?]
at 
org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92)
 [stormjar.jar:?]
at 
org.apache.metron.guava.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
 [stormjar.jar:?]
at 
org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:84)
 [stormjar.jar:?]
at 
org.apache.curator.framework.recipes.cache.TreeCache.callListeners(TreeCache.java:678)
 [stormjar.jar:?]
at

[jira] [Commented] (METRON-1466) The Elastic Search index are not getting generated in the latest metron deployments for all the topologies

2018-03-02 Thread Michael Miklavcic (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383908#comment-16383908
 ] 

Michael Miklavcic commented on METRON-1466:
---

Nevermind, this isn't even getting to the indexing topology.

> The Elastic Search index are not getting generated in the latest metron 
> deployments for all the topologies
> --
>
> Key: METRON-1466
> URL: https://issues.apache.org/jira/browse/METRON-1466
> Project: Metron
>  Issue Type: Bug
> Environment: Metron secured cluster 
>Reporter: Mohan
>Priority: Major
> Attachments: bro_worker.log, enrichmnts_worker.log, 
> indexing_worker.log
>
>
> The Elastic Search index are not getting generated in the latest metron 
> deployments for all the topologies , 
> I see that the data is getting published to the respective topics and also 
> getting enriched .
> below are the console consumer output for topics "bro" and "enrichments" I 
> dont get anything for "indexing" topic .
> {code:java}
> ./kafka-console-consumer.sh --zookeeper 
> ctr-e138-1518143905142-61778-01-02.hwx.site:2181 --security-protocol 
> PLAINTEXTSASL --topic bro 
> {"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
>  (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 
> 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 
> 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
>  
> {"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
>  (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 
> 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 
> 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
>  
> {"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
>  (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 
> 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 
> 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
>  
> {"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
>  (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 
> 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 
> 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
>  
> {"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
>  (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 
> 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 
> 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
>  ./kafka-console-consumer.sh --zookeeper 
> ctr-e138-1518143905142-61778-01-02.hwx.site:2181 --security-protocol 
> PLAINTEXTSASL --topic enrichments 
>

[jira] [Commented] (METRON-1466) The Elastic Search index are not getting generated in the latest metron deployments for all the topologies

2018-03-02 Thread Michael Miklavcic (JIRA)


[ 
https://issues.apache.org/jira/browse/METRON-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383894#comment-16383894
 ] 

Michael Miklavcic commented on METRON-1466:
---

Can you provide/paste the index listing for ES via the links in Ambari?

Also, can you check the Ambari service install logs to see if there are any 
errors there? If you can upload that as well that would be helpful.

> The Elastic Search index are not getting generated in the latest metron 
> deployments for all the topologies
> --
>
> Key: METRON-1466
> URL: https://issues.apache.org/jira/browse/METRON-1466
> Project: Metron
>  Issue Type: Bug
> Environment: Metron secured cluster 
>Reporter: Mohan
>Priority: Major
> Attachments: bro_worker.log, enrichmnts_worker.log, 
> indexing_worker.log
>
>
> The Elastic Search index are not getting generated in the latest metron 
> deployments for all the topologies , 
> I see that the data is getting published to the respective topics and also 
> getting enriched .
> below are the console consumer output for topics "bro" and "enrichments" I 
> dont get anything for "indexing" topic .
> {code:java}
> ./kafka-console-consumer.sh --zookeeper 
> ctr-e138-1518143905142-61778-01-02.hwx.site:2181 --security-protocol 
> PLAINTEXTSASL --topic bro 
> {"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
>  (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 
> 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 
> 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
>  
> {"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
>  (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 
> 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 
> 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
>  
> {"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
>  (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 
> 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 
> 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
>  
> {"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
>  (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 
> 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 
> 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
>  
> {"http":{"id.orig_p":49210,"status_code":200,"method":"GET","request_body_len":0,"id.resp_p":80,"uri":"\/picture.php?k=11iqmfg","tags":[],"uid":"CRDObQRKAmoHCQq1a","referrer":"http:\/\/7oqnsnzwwnm6zb7y.gigapaysun.com\/11iQmfg","resp_mime_types":["image\/png"],"trans_depth":3,"host":"7oqnsnzwwnm6zb7y.gigapaysun.com","status_msg":"OK","id.orig_h":"12.172.138.158","response_body_len":1823,"user_agent":"Mozilla\/4.0
>  (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident\/4.0; SLCC2; .NET CLR 
> 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 
> 6.0)","ts":1.469686182899762E9,"id.resp_h":"13.163.121.204","resp_fuids":["FQbZZfax5TLVX6M42"]}}
>  ./kafka-console-consumer.sh --zookeeper 
> ctr-e138-1518143905142-61778-01-02.hwx.site:2181

[jira] [Commented] (METRON-1466) The Elastic Search index are not getting generated in the latest metron deployments for all the topologies

[jira] [Commented] (METRON-1466) The Elastic Search index are not getting generated in the latest metron deployments for all the topologies

[jira] [Commented] (METRON-1466) The Elastic Search index are not getting generated in the latest metron deployments for all the topologies

3 matches

Site Navigation

Mail list logo

Footer information