That happens when you don't have the zookeeper url configured correctly. Can you check the contents of the /etc/sysconfig/metron file on the Metron host?
On Wed, May 17, 2017 at 1:36 PM, Laurens Vets <[email protected]> wrote: > For testing purposes, I decided to spin up the default Metron AWS config. > This resulted in a hang from ansible here: > > TASK [librdkafka : include] ****************************** > ********************** > task path: /home/laurens/SAPSource/metron/metron-deployment/roles/ > librdkafka/tasks/main.yml:18 > included: /home/laurens/SAPSource/metron/metron-deployment/roles/ > librdkafka/tasks/dependencies.yml for ec2-34-210-194-189.us-west-2.c > ompute.amazonaws.com > > TASK [librdkafka : Install prerequisites] ****************************** > ******** > task path: /home/laurens/SAPSource/metron/metron-deployment/roles/ > librdkafka/tasks/dependencies.yml:18 > <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> ESTABLISH CONNECTION > FOR USER: centos on PORT 22 TO ec2-34-210-194-189.us-west-2.c > ompute.amazonaws.com > /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141: > FutureWarning: CTR mode needs counter parameter, not IV > self._cipher = factory.new(key, *args, **kwargs) > <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC ( umask 22 && > mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508 > )" && echo "$( echo > $HOME/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508 > )" ) > <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> PUT /tmp/tmpwnH61y > TO /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/yum > <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC /bin/sh -c > 'sudo -H -S -n -u root /bin/sh -c '"'"'echo > BECOME-SUCCESS-rmswjjyhfdywqvwtvqwcmbsqpsbohvxh; > LANG=en_CA.UTF-8 LC_ALL=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8 /usr/bin/python > -tt /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/yum; > rm -rf "/home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/" > > /dev/null 2>&1'"'"'' > > Looking in the machine logs, I see the following for Kafka and Metron REST: > > Kafka: > [2017-05-17 17:03:14,831] INFO KafkaConfig values: > advertised.host.name = null > metric.reporters = [] > quota.producer.default = 9223372036854775807 > offsets.topic.num.partitions = 50 > log.flush.interval.messages = 9223372036854775807 > auto.create.topics.enable = true > controller.socket.timeout.ms = 30000 > log.flush.interval.ms = null > principal.builder.class = class org.apache.kafka.common.securi > ty.auth.DefaultPrincipalBuilder > replica.socket.receive.buffer.bytes = 65536 > min.insync.replicas = 1 > replica.fetch.wait.max.ms = 500 > num.recovery.threads.per.data.dir = 1 > ssl.keystore.type = JKS > sasl.mechanism.inter.broker.protocol = GSSAPI > default.replication.factor = 1 > ssl.truststore.password = null > log.preallocate = false > sasl.kerberos.principal.to.local.rules = [DEFAULT] > fetch.purgatory.purge.interval.requests = 10000 > ssl.endpoint.identification.algorithm = null > replica.socket.timeout.ms = 30000 > message.max.bytes = 1000000 > num.io.threads = 8 > offsets.commit.required.acks = -1 > log.flush.offset.checkpoint.interval.ms = 60000 > delete.topic.enable = false > quota.window.size.seconds = 1 > ssl.truststore.type = JKS > offsets.commit.timeout.ms = 5000 > quota.window.num = 11 > zookeeper.connect = ec2-34-223-200-113.us-west-2.c > ompute.amazonaws.com:2181 > authorizer.class.name = > num.replica.fetchers = 1 > log.retention.ms = null > log.roll.jitter.hours = 0 > log.cleaner.enable = true > offsets.load.buffer.size = 5242880 > log.cleaner.delete.retention.ms = 86400000 > ssl.client.auth = none > controlled.shutdown.max.retries = 3 > queued.max.requests = 500 > offsets.topic.replication.factor = 3 > log.cleaner.threads = 1 > sasl.kerberos.service.name = null > sasl.kerberos.ticket.renew.jitter = 0.05 > socket.request.max.bytes = 104857600 > ssl.trustmanager.algorithm = PKIX > zookeeper.session.timeout.ms = 30000 > log.retention.bytes = -1 > log.message.timestamp.type = CreateTime > sasl.kerberos.min.time.before.relogin = 60000 > zookeeper.set.acl = false > connections.max.idle.ms = 600000 > offsets.retention.minutes = 86400000 > replica.fetch.backoff.ms = 1000 > inter.broker.protocol.version = 0.10.0-IV1 > log.retention.hours = 168 > num.partitions = 1 > broker.id.generation.enable = true > listeners = PLAINTEXT://ec2-34-209-53-166. > us-west-2.compute.amazonaws.com:6667 > ssl.provider = null > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] > log.roll.ms = null > log.flush.scheduler.interval.ms = 9223372036854775807 > ssl.cipher.suites = null > log.index.size.max.bytes = 10485760 > ssl.keymanager.algorithm = SunX509 > security.inter.broker.protocol = PLAINTEXT > replica.fetch.max.bytes = 1048576 > advertised.port = null > log.cleaner.dedupe.buffer.size = 134217728 > replica.high.watermark.checkpoint.interval.ms = 5000 > log.cleaner.io.buffer.size = 524288 > sasl.kerberos.ticket.renew.window.factor = 0.8 > zookeeper.connection.timeout.ms = 25000 > controlled.shutdown.retry.backoff.ms = 5000 > log.roll.hours = 168 > log.cleanup.policy = delete > host.name = > log.roll.jitter.ms = null > max.connections.per.ip = 2147483647 > offsets.topic.segment.bytes = 104857600 > background.threads = 10 > quota.consumer.default = 9223372036854775807 > request.timeout.ms = 30000 > log.message.format.version = 0.10.0-IV1 > log.index.interval.bytes = 4096 > log.dir = /tmp/kafka-logs > log.segment.bytes = 1073741824 > log.cleaner.backoff.ms = 15000 > offset.metadata.max.bytes = 4096 > ssl.truststore.location = null > group.max.session.timeout.ms = 300000 > ssl.keystore.password = null > zookeeper.sync.time.ms = 2000 > port = 6667 > log.retention.minutes = null > log.segment.delete.delay.ms = 60000 > log.dirs = /data2/kafka-log > controlled.shutdown.enable = true > compression.type = producer > max.connections.per.ip.overrides = > log.message.timestamp.difference.max.ms = 9223372036854775807 > sasl.kerberos.kinit.cmd = /usr/bin/kinit > log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308 > auto.leader.rebalance.enable = true > leader.imbalance.check.interval.seconds = 300 > log.cleaner.min.cleanable.ratio = 0.5 > replica.lag.time.max.ms = 10000 > num.network.threads = 3 > ssl.key.password = null > reserved.broker.max.id = 1000 > metrics.num.samples = 2 > socket.send.buffer.bytes = 102400 > ssl.protocol = TLS > socket.receive.buffer.bytes = 102400 > ssl.keystore.location = null > replica.fetch.min.bytes = 1 > broker.rack = null > unclean.leader.election.enable = true > sasl.enabled.mechanisms = [GSSAPI] > group.min.session.timeout.ms = 6000 > log.cleaner.io.buffer.load.factor = 0.9 > offsets.retention.check.interval.ms = 600000 > producer.purgatory.purge.interval.requests = 10000 > metrics.sample.window.ms = 30000 > broker.id = -1 > offsets.topic.compression.codec = 0 > log.retention.check.interval.ms = 300000 > advertised.listeners = null > leader.imbalance.per.broker.percentage = 10 > (kafka.server.KafkaConfig) > [2017-05-17 17:03:14,990] INFO starting (kafka.server.KafkaServer) > [2017-05-17 17:03:14,998] INFO Connecting to zookeeper on > ec2-34-223-200-113.us-west-2.compute.amazonaws.com:2181 > (kafka.server.KafkaServer) > [2017-05-17 17:03:40,740] FATAL Fatal error during KafkaServer startup. > Prepare to shutdown (kafka.server.KafkaServer) > org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to > zookeeper server within timeout: 25000 > at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232) > at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156) > at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130) > at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils. > scala:75) > at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57) > at kafka.server.KafkaServer.initZk(KafkaServer.scala:294) > at kafka.server.KafkaServer.startup(KafkaServer.scala:180) > at kafka.server.KafkaServerStartable.startup(KafkaServerStartab > le.scala:37) > at kafka.Kafka$.main(Kafka.scala:67) > at kafka.Kafka.main(Kafka.scala) > [2017-05-17 17:03:40,740] INFO shutting down (kafka.server.KafkaServer) > [2017-05-17 17:03:40,747] INFO shut down completed > (kafka.server.KafkaServer) > [2017-05-17 17:03:40,748] FATAL Fatal error during KafkaServerStartable > startup. Prepare to shutdown (kafka.server.KafkaServerStartable) > org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to > zookeeper server within timeout: 25000 > at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232) > at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156) > at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130) > at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils. > scala:75) > at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57) > at kafka.server.KafkaServer.initZk(KafkaServer.scala:294) > at kafka.server.KafkaServer.startup(KafkaServer.scala:180) > at kafka.server.KafkaServerStartable.startup(KafkaServerStartab > le.scala:37) > at kafka.Kafka$.main(Kafka.scala:67) > at kafka.Kafka.main(Kafka.scala) > [2017-05-17 17:03:40,764] INFO shutting down (kafka.server.KafkaServer) > > Manually starting Kafka via Ambari fixes this... However, Metron REST > still doesn't work: > > . ____ _ __ _ _ > /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ > ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ > \\/ ___)| |_)| | | | | || (_| | ) ) ) ) > ' |____| .__|_| |_|_| |_\__, | / / / / > =========|_|==============|___/=/_/_/_/ > :: Spring Boot :: (v1.4.1.RELEASE) > > 2017-05-17T17:03:52.406 ERROR [org.springframework.boot.SpringApplication] > - Application startup failed > org.springframework.beans.factory.UnsatisfiedDependencyException: Error > creating bean with name 'kafkaServiceImpl' defined in URL > [jar:file:/usr/metron/0.4.0/lib/metron-rest-0.4.0.jar!/BOOT- > INF/classes!/org/apache/metron/rest/service/impl/KafkaServiceImpl.class]: > Unsatisfied dependency expressed through constructor parameter 0; nested > exception is org.springframework.beans.factory.UnsatisfiedDependencyException: > Error creating bean with name 'kafkaConfig': Unsatisfied dependency > expressed through field 'zkClient'; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating > bean with name 'zkClient' defined in class path resource > [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean instantiation > via factory method failed; nested exception is > org.springframework.beans.BeanInstantiationException: Failed to > instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' > threw exception; nested exception is > org.I0Itec.zkclient.exception.ZkTimeoutException: > Unable to connect to zookeeper server within timeout: 10000 > at org.springframework.beans.factory.support.ConstructorResolve > r.createArgumentArray(ConstructorResolver.java:749) > at org.springframework.beans.factory.support.ConstructorResolve > r.autowireConstructor(ConstructorResolver.java:189) > at org.springframework.beans.factory.support.AbstractAutowireCa > pableBeanFactory.autowireConstructor(AbstractAu > towireCapableBeanFactory.java:1148) > at org.springframework.beans.factory.support.AbstractAutowireCa > pableBeanFactory.createBeanInstance(AbstractAut > owireCapableBeanFactory.java:1051) > at org.springframework.beans.factory.support.AbstractAutowireCa > pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510) > at org.springframework.beans.factory.support.AbstractAutowireCa > pableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) > at org.springframework.beans.factory.support.AbstractBeanFactor > y$1.getObject(AbstractBeanFactory.java:306) > at org.springframework.beans.factory.support.DefaultSingletonBe > anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) > at org.springframework.beans.factory.support.AbstractBeanFactor > y.doGetBean(AbstractBeanFactory.java:302) > at org.springframework.beans.factory.support.AbstractBeanFactor > y.getBean(AbstractBeanFactory.java:197) > at org.springframework.beans.factory.support.DefaultListableBea > nFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751) > at org.springframework.context.support.AbstractApplicationConte > xt.finishBeanFactoryInitialization(AbstractApplicationContext.java:861) > at org.springframework.context.support.AbstractApplicationConte > xt.refresh(AbstractApplicationContext.java:541) > at org.springframework.boot.context.embedded.EmbeddedWebApplica > tionContext.refresh(EmbeddedWebApplicationContext.java:122) > at org.springframework.boot.SpringApplication.refresh(SpringApp > lication.java:761) > at org.springframework.boot.SpringApplication.refreshContext(Sp > ringApplication.java:371) > at org.springframework.boot.SpringApplication.run(SpringApplica > tion.java:315) > at org.springframework.boot.SpringApplication.run(SpringApplica > tion.java:1186) > at org.springframework.boot.SpringApplication.run(SpringApplica > tion.java:1175) > at org.apache.metron.rest.MetronRestApplication.main(MetronRest > Application.java:27) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce > ssorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe > thodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.springframework.boot.loader.MainMethodRunner.run(MainMet > hodRunner.java:48) > at org.springframework.boot.loader.Launcher.launch(Launcher. > java:87) > at org.springframework.boot.loader.Launcher.launch(Launcher. > java:50) > at org.springframework.boot.loader.PropertiesLauncher.main( > PropertiesLauncher.java:525) > Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: > Error creating bean with name 'kafkaConfig': Unsatisfied dependency > expressed through field 'zkClient'; nested exception is > org.springframework.beans.factory.BeanCreationException: Error creating > bean with name 'zkClient' defined in class path resource > [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean instantiation > via factory method failed; nested exception is > org.springframework.beans.BeanInstantiationException: Failed to > instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' > threw exception; nested exception is > org.I0Itec.zkclient.exception.ZkTimeoutException: > Unable to connect to zookeeper server within timeout: 10000 > at org.springframework.beans.factory.annotation.AutowiredAnnota > tionBeanPostProcessor$AutowiredFieldElement.inject(A > utowiredAnnotationBeanPostProcessor.java:569) > at org.springframework.beans.factory.annotation.InjectionMetada > ta.inject(InjectionMetadata.java:88) > at org.springframework.beans.factory.annotation.AutowiredAnnota > tionBeanPostProcessor.postProcessPropertyValues(AutowiredAnn > otationBeanPostProcessor.java:349) > at org.springframework.beans.factory.support.AbstractAutowireCa > pableBeanFactory.populateBean(AbstractAutowireCapableBeanFac > tory.java:1219) > at org.springframework.beans.factory.support.AbstractAutowireCa > pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543) > at org.springframework.beans.factory.support.AbstractAutowireCa > pableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) > at org.springframework.beans.factory.support.AbstractBeanFactor > y$1.getObject(AbstractBeanFactory.java:306) > at org.springframework.beans.factory.support.DefaultSingletonBe > anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) > at org.springframework.beans.factory.support.AbstractBeanFactor > y.doGetBean(AbstractBeanFactory.java:302) > at org.springframework.beans.factory.support.AbstractBeanFactor > y.getBean(AbstractBeanFactory.java:197) > at org.springframework.beans.factory.support.ConstructorResolve > r.instantiateUsingFactoryMethod(ConstructorResolver.java:372) > at org.springframework.beans.factory.support.AbstractAutowireCa > pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow > ireCapableBeanFactory.java:1128) > at org.springframework.beans.factory.support.AbstractAutowireCa > pableBeanFactory.createBeanInstance(AbstractAut > owireCapableBeanFactory.java:1023) > at org.springframework.beans.factory.support.AbstractAutowireCa > pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510) > at org.springframework.beans.factory.support.AbstractAutowireCa > pableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) > at org.springframework.beans.factory.support.AbstractBeanFactor > y$1.getObject(AbstractBeanFactory.java:306) > at org.springframework.beans.factory.support.DefaultSingletonBe > anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) > at org.springframework.beans.factory.support.AbstractBeanFactor > y.doGetBean(AbstractBeanFactory.java:302) > at org.springframework.beans.factory.support.AbstractBeanFactor > y.getBean(AbstractBeanFactory.java:202) > at org.springframework.beans.factory.config.DependencyDescripto > r.resolveCandidate(DependencyDescriptor.java:207) > at org.springframework.beans.factory.support.DefaultListableBea > nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128) > at org.springframework.beans.factory.support.DefaultListableBea > nFactory.resolveDependency(DefaultListableBeanFactory.java:1056) > at org.springframework.beans.factory.support.ConstructorResolve > r.resolveAutowiredArgument(ConstructorResolver.java:835) > at org.springframework.beans.factory.support.ConstructorResolve > r.createArgumentArray(ConstructorResolver.java:741) > ... 27 more > Caused by: org.springframework.beans.factory.BeanCreationException: Error > creating bean with name 'zkClient' defined in class path resource > [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean instantiation > via factory method failed; nested exception is > org.springframework.beans.BeanInstantiationException: Failed to > instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' > threw exception; nested exception is > org.I0Itec.zkclient.exception.ZkTimeoutException: > Unable to connect to zookeeper server within timeout: 10000 > at org.springframework.beans.factory.support.ConstructorResolve > r.instantiateUsingFactoryMethod(ConstructorResolver.java:599) > at org.springframework.beans.factory.support.AbstractAutowireCa > pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow > ireCapableBeanFactory.java:1128) > at org.springframework.beans.factory.support.AbstractAutowireCa > pableBeanFactory.createBeanInstance(AbstractAut > owireCapableBeanFactory.java:1023) > at org.springframework.beans.factory.support.AbstractAutowireCa > pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510) > at org.springframework.beans.factory.support.AbstractAutowireCa > pableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) > at org.springframework.beans.factory.support.AbstractBeanFactor > y$1.getObject(AbstractBeanFactory.java:306) > at org.springframework.beans.factory.support.DefaultSingletonBe > anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) > at org.springframework.beans.factory.support.AbstractBeanFactor > y.doGetBean(AbstractBeanFactory.java:302) > at org.springframework.beans.factory.support.AbstractBeanFactor > y.getBean(AbstractBeanFactory.java:202) > at org.springframework.beans.factory.config.DependencyDescripto > r.resolveCandidate(DependencyDescriptor.java:207) > at org.springframework.beans.factory.support.DefaultListableBea > nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128) > at org.springframework.beans.factory.support.DefaultListableBea > nFactory.resolveDependency(DefaultListableBeanFactory.java:1056) > at org.springframework.beans.factory.annotation.AutowiredAnnota > tionBeanPostProcessor$AutowiredFieldElement.inject(A > utowiredAnnotationBeanPostProcessor.java:566) > ... 50 more > Caused by: org.springframework.beans.BeanInstantiationException: Failed > to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' > threw exception; nested exception is > org.I0Itec.zkclient.exception.ZkTimeoutException: > Unable to connect to zookeeper server within timeout: 10000 > at org.springframework.beans.factory.support.SimpleInstantiatio > nStrategy.instantiate(SimpleInstantiationStrategy.java:189) > at org.springframework.beans.factory.support.ConstructorResolve > r.instantiateUsingFactoryMethod(ConstructorResolver.java:588) > ... 62 more > Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to > connect to zookeeper server within timeout: 10000 > at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232) > at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156) > at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130) > at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:97) > at org.apache.metron.rest.config.ZookeeperConfig.zkClient(Zooke > eperConfig.java:52) > at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr > ingCGLIB$$6e0c6742.CGLIB$zkClient$1(<generated>) > at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr > ingCGLIB$$6e0c6742$$FastClassBySpringCGLIB$$3219d861.invoke(<generated>) > at org.springframework.cglib.proxy.MethodProxy.invokeSuper(Meth > odProxy.java:228) > at org.springframework.context.annotation.ConfigurationClassEnh > ancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:356) > at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr > ingCGLIB$$6e0c6742.zkClient(<generated>) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce > ssorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe > thodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.springframework.beans.factory.support.SimpleInstantiatio > nStrategy.instantiate(SimpleInstantiationStrategy.java:162) > ... 63 more > > Any ideas on what might be going or why the timeout triggers? > > > On 2017-05-17 08:31, Laurens Vets wrote: > >> Hello list, >> >> I'm trying to spin up Metron in EC2 for testing purposes. I made the >> following changes in playbook.yml so that I'm left with 5 EC2 >> instances instead of 10: >> >> <-- SNIP --> >> tasks: >> - include: tasks/create-keypair.yml >> - include: tasks/create-vpc.yml >> - include: tasks/create-open-inbound-security-group.yml >> - include: tasks/create-open-outbound-security-group.yml >> - include: tasks/create-hosts.yml host_count=1 >> host_type=sensors,ambari_master,ec2,monit >> # - include: tasks/create-hosts.yml host_count=4 >> host_type=ambari_slave,ec2 >> - include: tasks/create-hosts.yml host_count=1 >> host_type=pcap_server,monit,ec2 >> - include: tasks/create-hosts.yml host_count=1 >> host_type=ambari_slave,enrichment,metron,ec2,zeppelin >> # - include: tasks/create-hosts.yml host_count=2 >> host_type=ambari_slave,search,ec2 >> - include: tasks/create-hosts.yml host_count=1 >> host_type=ambari_slave,search,ec2 >> - include: tasks/create-hosts.yml host_count=1 >> host_type=ambari_slave,web,ec2 >> <-- END SNIP --> >> >> I also had to change the line "ansible-playbook -vvv -i ec2.py >> playbook.yml \ >> --skip-tags="quick_dev,sensor-stubs" \ >> --extra-vars="env=$ENV" \ >> $EXTRA_ARGS" >> >> to "ansible-playbook -vvv -i ec2.py playbook.yml \ >> --skip-tags="quick_dev,sensor-stubs" \ >> --extra-vars="env=$ENV" \ >> $EXTRA_ARGS -c paramiko" >> in run.sh otherwise ssh would time out. >> >> However, Ansible errors out with: >> >> 2017-05-16 18:02:50,615 p=5362 u=laurens | TASK [ambari_config : >> Deploy cluster with Ambari; >> http://ec2-35-164-26-255.us-west-2.compute.amazonaws.com:8080] *** >> 2017-05-16 18:02:50,615 p=5362 u=laurens | task path: >> /home/laurens/SAPSource/metron/metron-deployment/roles/ >> ambari_config/tasks/main.yml:36 >> 2017-05-16 18:02:52,660 p=5362 u=laurens | fatal: >> [ec2-35-164-26-255.us-west-2.compute.amazonaws.com]: FAILED! => >> {"changed": false, "failed": true, "invocation": {"module_args": >> {"blueprint_name": "metron_blueprint", "blueprint_var": {"groups": >> [{"cardinality": 1, "components": [{"name": "NAMENODE"}, {"name": >> "SECONDARY_NAMENODE"}, {"name": "RESOURCEMANAGER"}, {"name": >> "HISTORYSERVER"}, {"name": "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, >> {"name": "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name": >> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name": "ES_SLAVE"}], >> "configuration": [], "name": "master_1"}, {"cardinality": 1, >> "components": [{"name": "ZOOKEEPER_SERVER"}, {"name": "NIMBUS"}, >> {"name": "STORM_UI_SERVER"}, {"name": "DRPC_SERVER"}, {"name": >> "SPARK_JOBHISTORYSERVER"}, {"name": "HBASE_MASTER"}, {"name": >> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name": >> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name": >> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name": >> "APP_TIMELINE_SERVER"}, {"name": "ES_SLAVE"}], "configuration": [], >> "name": "master_2"}, {"cardinality": 1, "components": [{"name": >> "METRON_INDEXING"}, {"name": "METRON_ENRICHMENT_MASTER"}, {"name": >> "METRON_PARSERS"}, {"name": "METRON_REST"}, {"name": "DATANODE"}, >> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name": >> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name": >> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name": >> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name": >> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [], >> "name": "metron"}, {"cardinality": 1, "components": [{"name": >> "KIBANA_MASTER"}, {"name": "ES_MASTER"}], "configuration": [], "name": >> "web"}, {"cardinality": "1+", "components": [{"name": "DATANODE"}, >> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name": >> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name": >> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name": >> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name": >> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [], >> "name": "slaves"}], "required_configurations": [{"metron-env": >> {"es_hosts": >> "ec2-35-164-136-88.us-west-2.compute.amazonaws.com,ec2-34-20 >> 8-94-45.us-west-2.compute.amazonaws.com", >> "metron_jdbc_driver": "org.h2.Driver", "metron_jdbc_password": "root", >> "metron_jdbc_platform": "h2", "metron_jdbc_url": >> "jdbc:h2:file:~/metrondb", "metron_jdbc_username": "root", >> "storm_rest_addr": >> "ec2-34-208-94-45.us-west-2.compute.amazonaws.com:8744", >> "zeppelin_server_url": >> "ec2-34-210-151-29.us-west-2.compute.amazonaws.com"}}, {"kibana-env": >> {"kibana_default_application": "dashboard/Metron-Dashboard", >> "kibana_es_url": >> "http://ec2-35-164-136-88.us-west-2.compute.amazonaws.com:9200";, >> "kibana_log_dir": "/var/log/kibana", "kibana_pid_dir": >> "/var/run/kibana", "kibana_server_port": 5000}}], "stack_name": "HDP", >> "stack_version": "2.5"}, "cluster_name": "metron", "cluster_state": >> "present", "configurations": [{"zoo.cfg": {"dataDir": >> "/data1/hadoop/zookeeper"}}, {"hadoop-env": {"dtnode_heapsize": 1024, >> "namenode_heapsize": 2048}}, {"hbase-env": {"hbase_master_heapsize": >> 1024, "hbase_regionserver_heapsize": 1024}}, {"hdfs-site": >> {"dfs.datanode.data.dir": >> "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data", >> "dfs.journalnode.edits.dir": "/data1/hadoop/hdfs/journalnode", >> "dfs.namenode.checkpoint.dir": "/data1/hadoop/hdfs/namesecondary", >> "dfs.namenode.name.dir": "/data1/hadoop/hdfs/namenode"}}, >> {"mapred-site": {"mapreduce.jobhistory.recovery.store.leveldb.path": >> "/data1/hadoop/mapreduce/jhs", "mapreduce.map.memory.mb": 1229, >> "mapreduce.reduce.memory.mb": 1229}}, {"yarn-site": >> {"yarn.nodemanager.local-dirs": "/data1/hadoop/yarn/local", >> "yarn.nodemanager.log-dirs": "/data1/hadoop/yarn/log", >> "yarn.nodemanager.resource.memory-mb": 4096, >> "yarn.timeline-service.leveldb-state-store.path": >> "/data1/hadoop/yarn/timeline", >> "yarn.timeline-service.leveldb-timeline-store.path": >> "/data1/hadoop/yarn/timeline"}}, {"storm-site": {"storm.local.dir": >> "/data1/hadoop/storm", "supervisor.slots.ports": "[6700, 6701, 6702, >> 6703]", "topology.classpath": "/etc/hbase/conf:/etc/hadoop/conf"}}, >> {"kafka-broker": {"log.dirs": "/data2/kafka-log"}}, {"metron-env": >> {"parsers": "bro,snort,yaf"}}, {"elastic-site": >> {"gateway_recover_after_data_nodes": 1, "index_number_of_replicas": 1, >> "index_number_of_shards": 2, "network_host": "[ _local_, _eth0_ ]", >> "zen_discovery_ping_unicast_hosts": "[ >> ec2-35-164-136-88.us-west-2.compute.amazonaws.com, >> ec2-34-208-94-45.us-west-2.compute.amazonaws.com ]"}}], "host": >> "ec2-35-164-26-255.us-west-2.compute.amazonaws.com", "password": >> "admin", "port": 8080, "username": "admin", "wait_for_complete": >> true}, "module_name": "ambari_cluster_state"}, "msg": "Ambari client >> exception occurred: Could not create cluster: request code 400, >> request message {\n \"status\" : 400,\n \"message\" : >> \"Invalid Cluster Creation Template: >> org.apache.ambari.server.topology.InvalidTopologyTemplateException: >> Host group 'slaves' must contain at least one 'hosts/fqdn' or a >> 'host_count' value\"\n}"} >> >> PLAY RECAP ************************************************************ >> ********* >> ec2-34-208-94-45.us-west-2.compute.amazonaws.com : ok=49 changed=27 >> unreachable=0 failed=0 >> ec2-34-209-224-190.us-west-2.compute.amazonaws.com : ok=23 changed=8 >> unreachable=0 failed=0 >> ec2-34-210-151-29.us-west-2.compute.amazonaws.com : ok=58 changed=31 >> unreachable=0 failed=0 >> ec2-35-164-136-88.us-west-2.compute.amazonaws.com : ok=49 changed=27 >> unreachable=0 failed=0 >> ec2-35-164-26-255.us-west-2.compute.amazonaws.com : ok=54 changed=28 >> unreachable=0 failed=1 >> localhost : ok=29 changed=12 unreachable=0 >> failed=0 >> >> Any idea what might be going on? >> >> (Don't worry about the URLs, the instances have been terminated) >> >
