[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/907
  
+1
I ran it up on 2 separate Macs, one running Ansible 2.2.2.0 and the other 
2.0.0.2. Both spun up without issue today.


---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/907
  
+1 Ran it up with all of our fixes in the Ubuntu dev environment.  Works 
great.  Thanks!


---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/907
  
I also just merged #913 so I will run this up just to be sure the whole 
package is working in the Ubuntu dev environment.


---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/907
  
@nickwallen Awesome, your PR is merged in here and I'll spin it up one last 
time to verify that it works for 2.0.0.2.

@anandsubbu Awesome, thanks!


---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/907
  
Hi @cestella , with the latest commit having the normalized producer, I was 
able to spin up a 12-node Centos 7 cluster, install with HDP 2.6 base and 
kerberize as well without issues. Ran the standard squid sensor test and was 
able to see events in the Alerts UI.

+1 from my end.


---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/907
  
FYI @cestella I submitted a fix against your PR branch that should address 
the issue with the embedded handlebars in the Ambari response. 


---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/907
  
For 2, that's the same error @mmiklavc hit on centos.  It's very 
interesting that I'm not hitting it.


---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/907
  
For (2), the problem is that the HDFS configuration in Ambari has embedded 
'mustache handlebars' (see `{{major_stack_version}}`) that when returned from 
an API call confuses Ansible.  We need some way to strip that out or ignore it.

![screen shot 2018-01-29 at 2 31 44 
pm](https://user-images.githubusercontent.com/2475409/35530186-5072f3ae-0501-11e8-8bd9-93096da51f02.png)



---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/907
  
Running this up on Ubuntu, I ran into two issues.

(1) JDK Install Failed on Ubuntu - I do not think this was caused by this 
PR.  This is something that I should have hit in #903, but the changes there 
never brought this bug to light.  Fortunately, it is an easy fix and I will 
open it as a separate PR.

(2) After the Ambari deployment completes, it begins to install the 
sensors.  It queries Ambari for information to configure the sensors and in one 
of those queries it hits this problem.  I am not sure exactly what the problem 
is yet.
```
TASK [ambari_gather_facts : Ask Ambari: hdfs_url] 
**
ok: [node1]

TASK [ambari_gather_facts : set_fact] 
**
fatal: [node1]: FAILED! =>{  
   "failed":true,
   "msg":"the field 'args' has an invalid value, which appears to include a 
variable that is undefined. The error was: {u'status': 200, u'content_type': 
u'text/plain', u'set_cookie': 
u'AMBARISESSIONID=5tlv0y7btgc24krm5ugdpdgj;Path=/;HttpOnly', u'expires': u'Thu, 
01 Jan 1970 00:00:00 GMT', u'vary': u'Accept-Encoding, User-Agent', u'user': 
u'admin', u'pragma': u'no-cache', u'x_frame_options': u'DENY', 
u'x_xss_protection': u'1; mode=block', u'url': 
u'http://node1:8080/api/v1/clusters/metron_cluster/configurations?type=core-site=TOPOLOGY_RESOLVED',
 u'changed': False, u'x_content_type_options': u'nosniff', u'content': u'{\\n  
\"href\" : 
\"http://node1:8080/api/v1/clusters/metron_cluster/configurations?type=core-site=TOPOLOGY_RESOLVED\",\\n
  \"items\" : [\\n{\\n  \"href\" : 
\"http://node1:8080/api/v1/clusters/metron_cluster/configurations?type=core-site=TOPOLOGY_RESOLVED\",\\n
  \"tag\" : \"TOPOLOGY_RESOLVED\",\\n  \"type\" : \"core-site\",\\n 
 \"version\
 " : 2,\\n  \"Config\" : {\\n\"cluster_name\" : 
\"metron_cluster\",\\n\"stack_id\" : \"HDP-2.6\"\\n  },\\n  
\"properties\" : {\\n\"fs.defaultFS\" : \"hdfs://node1:8020\",\\n   
 \"fs.trash.interval\" : \"360\",\\n
\"ha.failover-controller.active-standby-elector.zk.op.retries\" : \"120\",\\n   
 \"hadoop.custom-extensions.root\" : 
\"/hdp/ext/{{major_stack_version}}/hadoop\",\\n
\"hadoop.http.authentication.simple.anonymous.allowed\" : \"true\",\\n
\"hadoop.proxyuser.hbase.groups\" : \"*\",\\n
\"hadoop.proxyuser.hbase.hosts\" : \"*\",\\n
\"hadoop.security.auth_to_local\" : \"DEFAULT\",\\n
\"hadoop.security.authentication\" : \"simple\",\\n
\"hadoop.security.authorization\" : \"false\",\\n
\"hadoop.security.key.provider.path\" : \"\",\\n
\"io.compression.codecs\" : 
\"org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.Snapp
 yCodec\",\\n\"io.file.buffer.size\" : \"131072\",\\n
\"io.serializations\" : 
\"org.apache.hadoop.io.serializer.WritableSerialization\",\\n
\"ipc.client.connect.max.retries\" : \"50\",\\n
\"ipc.client.connection.maxidletime\" : \"3\",\\n
\"ipc.client.idlethreshold\" : \"8000\",\\n\"ipc.server.tcpnodelay\" : 
\"true\",\\n\"mapreduce.jobtracker.webinterface.trusted\" : 
\"false\",\\n\"net.topology.script.file.name\" : 
\"/etc/hadoop/conf/topology_script.py\"\\n  },\\n  
\"properties_attributes\" : {\\n\"final\" : {\\n  
\"fs.defaultFS\" : \"true\"\\n}\\n  }\\n}\\n  ]\\n}', 
u'connection': u'close', u'msg': u'OK (unknown bytes)', u'redirected': False, 
u'cache_control': u'no-store'}: 'major_stack_version' is undefined\n\nThe error 
appears to have been in 
'/Users/nallen/tmp/metron-pr907/metron-deployment/ansible/roles/ambari_gather_facts/tasks/main.yml':
 line 82, column 3, but may\nbe elsewhe
 re in the file depending on the exact syntax problem.\n\nThe offending line 
appears to be:\n\n\n- set_fact:\n  ^ here\n"
}
```




---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/907
  
I'm testing now as well for good measure since this is our main dev testing 
environment. We should probably take a little extra care with the breadth of 
changes in this and https://github.com/apache/metron/pull/903


---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/907
  
ok, I ran this guy up in kerberos and regular and tooled around a bit; 
ensured the alerts UI worked in both.


---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/907
  
Hi @cestella , I did a 12-node deploy on CentOS 7 with this patch. 
Post-kerberization, I noticed the following errors in Metron REST. Is this a 
related issue or a different one?

```
18/01/25 10:16:02 ERROR boot.SpringApplication: Application startup failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
creating bean with name 'alertController': Unsatisfied dependency expressed 
through field 'alertService'; nested exception is 
org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
creating bean with name 'alertServiceImpl' defined in URL 
[jar:file:/usr/metron/0.4.3/lib/metron-rest-0.4.3.jar!/org/apache/metron/rest/service/impl/AlertServiceImpl.class]:
 Unsatisfied dependency expressed through constructor parameter 0; nested 
exception is org.springframework.beans.factory.UnsatisfiedDependencyException: 
Error creating bean with name 'kafkaServiceImpl' defined in URL 
[jar:file:/usr/metron/0.4.3/lib/metron-rest-0.4.3.jar!/org/apache/metron/rest/service/impl/KafkaServiceImpl.class]:
 Unsatisfied dependency expressed through constructor parameter 2; nested 
exception is org.springframework.beans.factory.BeanCreationException: Error 
creating bean with name 'kafkaProducer' defined in class path resourc
 e [org/apache/metron/rest/config/KafkaConfig.class]: Bean instantiation via 
factory method failed; nested exception is 
org.springframework.beans.BeanInstantiationException: Failed to instantiate 
[org.apache.kafka.clients.producer.KafkaProducer]: Factory method 
'kafkaProducer' threw exception; nested exception is 
org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at 
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:569)
at 
org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at 
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:349)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1219)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at 
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at 
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at 
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at 
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at 
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751)
at 
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:861)
at 
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541)
at 
org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
at 
org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761)
at 
org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:371)
at 
org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at 
org.springframework.boot.SpringApplication.run(SpringApplication.java:1186)
at 
org.springframework.boot.SpringApplication.run(SpringApplication.java:1175)
at 
org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:29)
Caused by: 
org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
creating bean with name 'alertServiceImpl' defined in URL 
[jar:file:/usr/metron/0.4.3/lib/metron-rest-0.4.3.jar!/org/apache/metron/rest/service/impl/AlertServiceImpl.class]:
 Unsatisfied dependency expressed through constructor parameter 0; nested 
exception is org.springframework.beans.factory.UnsatisfiedDependencyException: 
Error creating bean with name 'kafkaServiceImpl' defined in URL 
[jar:file:/usr/metron/0.4.3/lib/metron-rest-0.4.3.jar!/org/apache/metron/rest/service/impl/KafkaServiceImpl.class]:
 Unsatisfied dependency expressed through constructor parameter 2; nested 
exception is org.springframework.beans.factory.BeanCreationException: Error 
creating bean with 

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/907
  
@Ali - I think the reason for Storm 1.1 is that it's the version that comes
with HDP 2.6.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_release-notes/content/comp_versions.html

On Wed, Jan 24, 2018 at 4:13 PM, Ali  wrote:

> @cestella  Would it be possible to target
> Storm 1.2 (which was released 2 days ago) as well?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[mraliagha]

Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/907
  
@cestella  Would it be possible to target Storm 1.2 (which was released 2 
days ago) as well?


---