I am not sure what the issue is. Nothing is jumping out at me. Weird issues can arise when the machine that you are running the Ansible deployment from has intermittent connectivity issues to the AWS data center.
Is Amazon's us-west-2 data center the one closest to you? You might try a different/closer data center. You can alter the data center here [1]. [1] https://github.com/apache/metron/blob/master/metron-deployment/amazon-ec2/conf/defaults.yml#L27-L39 Other than that, you could try standing up a small node in AWS and then use that to run the Metron deployment from. That always works much more smoothly. On Wed, May 17, 2017 at 4:41 PM, Laurens Vets <[email protected]> wrote: > Also, it's only 1 out of 10 hosts that failing it seems. > > > On 2017-05-17 13:25, Laurens Vets wrote: > >> When you say Metron host, do you mean the host I'm pushing from to ec2 >> or the master ambari server in ec2? >> >> On 2017-05-17 11:51, Ryan Merriman wrote: >> >>> That happens when you don't have the zookeeper url configured correctly. >>> Can you check the contents of the /etc/sysconfig/metron file on the >>> Metron >>> host? >>> >>> On Wed, May 17, 2017 at 1:36 PM, Laurens Vets <[email protected]> wrote: >>> >>> For testing purposes, I decided to spin up the default Metron AWS config. >>>> This resulted in a hang from ansible here: >>>> >>>> TASK [librdkafka : include] ****************************** >>>> ********************** >>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/ >>>> librdkafka/tasks/main.yml:18 >>>> included: /home/laurens/SAPSource/metron/metron-deployment/roles/ >>>> librdkafka/tasks/dependencies.yml for ec2-34-210-194-189.us-west-2.c >>>> ompute.amazonaws.com >>>> >>>> TASK [librdkafka : Install prerequisites] ****************************** >>>> ******** >>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/ >>>> librdkafka/tasks/dependencies.yml:18 >>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> ESTABLISH >>>> CONNECTION >>>> FOR USER: centos on PORT 22 TO ec2-34-210-194-189.us-west-2.c >>>> ompute.amazonaws.com >>>> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141: >>>> FutureWarning: CTR mode needs counter parameter, not IV >>>> self._cipher = factory.new(key, *args, **kwargs) >>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC ( umask 22 && >>>> mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp >>>> -1495041091.74-92163853889508 >>>> )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp >>>> -1495041091.74-92163853889508 >>>> )" ) >>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> PUT /tmp/tmpwnH61y >>>> TO /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853 >>>> 889508/yum >>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC /bin/sh -c >>>> 'sudo -H -S -n -u root /bin/sh -c '"'"'echo >>>> BECOME-SUCCESS-rmswjjyhfdywqvwtvqwcmbsqpsbohvxh; >>>> LANG=en_CA.UTF-8 LC_ALL=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8 >>>> /usr/bin/python >>>> -tt /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853 >>>> 889508/yum; >>>> rm -rf "/home/centos/.ansible/tmp/ansible-tmp-1495041091.74-9216385 >>>> 3889508/" >>>> > /dev/null 2>&1'"'"'' >>>> >>>> Looking in the machine logs, I see the following for Kafka and Metron >>>> REST: >>>> >>>> Kafka: >>>> [2017-05-17 17:03:14,831] INFO KafkaConfig values: >>>> advertised.host.name = null >>>> metric.reporters = [] >>>> quota.producer.default = 9223372036854775807 >>>> offsets.topic.num.partitions = 50 >>>> log.flush.interval.messages = 9223372036854775807 >>>> auto.create.topics.enable = true >>>> controller.socket.timeout.ms = 30000 >>>> log.flush.interval.ms = null >>>> principal.builder.class = class org.apache.kafka.common.securi >>>> ty.auth.DefaultPrincipalBuilder >>>> replica.socket.receive.buffer.bytes = 65536 >>>> min.insync.replicas = 1 >>>> replica.fetch.wait.max.ms = 500 >>>> num.recovery.threads.per.data.dir = 1 >>>> ssl.keystore.type = JKS >>>> sasl.mechanism.inter.broker.protocol = GSSAPI >>>> default.replication.factor = 1 >>>> ssl.truststore.password = null >>>> log.preallocate = false >>>> sasl.kerberos.principal.to.local.rules = [DEFAULT] >>>> fetch.purgatory.purge.interval.requests = 10000 >>>> ssl.endpoint.identification.algorithm = null >>>> replica.socket.timeout.ms = 30000 >>>> message.max.bytes = 1000000 >>>> num.io.threads = 8 >>>> offsets.commit.required.acks = -1 >>>> log.flush.offset.checkpoint.interval.ms = 60000 >>>> delete.topic.enable = false >>>> quota.window.size.seconds = 1 >>>> ssl.truststore.type = JKS >>>> offsets.commit.timeout.ms = 5000 >>>> quota.window.num = 11 >>>> zookeeper.connect = ec2-34-223-200-113.us-west-2.c >>>> ompute.amazonaws.com:2181 >>>> authorizer.class.name = >>>> num.replica.fetchers = 1 >>>> log.retention.ms = null >>>> log.roll.jitter.hours = 0 >>>> log.cleaner.enable = true >>>> offsets.load.buffer.size = 5242880 >>>> log.cleaner.delete.retention.ms = 86400000 >>>> ssl.client.auth = none >>>> controlled.shutdown.max.retries = 3 >>>> queued.max.requests = 500 >>>> offsets.topic.replication.factor = 3 >>>> log.cleaner.threads = 1 >>>> sasl.kerberos.service.name = null >>>> sasl.kerberos.ticket.renew.jitter = 0.05 >>>> socket.request.max.bytes = 104857600 >>>> ssl.trustmanager.algorithm = PKIX >>>> zookeeper.session.timeout.ms = 30000 >>>> log.retention.bytes = -1 >>>> log.message.timestamp.type = CreateTime >>>> sasl.kerberos.min.time.before.relogin = 60000 >>>> zookeeper.set.acl = false >>>> connections.max.idle.ms = 600000 >>>> offsets.retention.minutes = 86400000 >>>> replica.fetch.backoff.ms = 1000 >>>> inter.broker.protocol.version = 0.10.0-IV1 >>>> log.retention.hours = 168 >>>> num.partitions = 1 >>>> broker.id.generation.enable = true >>>> listeners = PLAINTEXT://ec2-34-209-53-166. >>>> us-west-2.compute.amazonaws.com:6667 >>>> ssl.provider = null >>>> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] >>>> log.roll.ms = null >>>> log.flush.scheduler.interval.ms = 9223372036854775807 >>>> ssl.cipher.suites = null >>>> log.index.size.max.bytes = 10485760 >>>> ssl.keymanager.algorithm = SunX509 >>>> security.inter.broker.protocol = PLAINTEXT >>>> replica.fetch.max.bytes = 1048576 >>>> advertised.port = null >>>> log.cleaner.dedupe.buffer.size = 134217728 >>>> replica.high.watermark.checkpoint.interval.ms = 5000 >>>> log.cleaner.io.buffer.size = 524288 >>>> sasl.kerberos.ticket.renew.window.factor = 0.8 >>>> zookeeper.connection.timeout.ms = 25000 >>>> controlled.shutdown.retry.backoff.ms = 5000 >>>> log.roll.hours = 168 >>>> log.cleanup.policy = delete >>>> host.name = >>>> log.roll.jitter.ms = null >>>> max.connections.per.ip = 2147483647 >>>> offsets.topic.segment.bytes = 104857600 >>>> background.threads = 10 >>>> quota.consumer.default = 9223372036854775807 >>>> request.timeout.ms = 30000 >>>> log.message.format.version = 0.10.0-IV1 >>>> log.index.interval.bytes = 4096 >>>> log.dir = /tmp/kafka-logs >>>> log.segment.bytes = 1073741824 >>>> log.cleaner.backoff.ms = 15000 >>>> offset.metadata.max.bytes = 4096 >>>> ssl.truststore.location = null >>>> group.max.session.timeout.ms = 300000 >>>> ssl.keystore.password = null >>>> zookeeper.sync.time.ms = 2000 >>>> port = 6667 >>>> log.retention.minutes = null >>>> log.segment.delete.delay.ms = 60000 >>>> log.dirs = /data2/kafka-log >>>> controlled.shutdown.enable = true >>>> compression.type = producer >>>> max.connections.per.ip.overrides = >>>> log.message.timestamp.difference.max.ms = 9223372036854775807 >>>> sasl.kerberos.kinit.cmd = /usr/bin/kinit >>>> log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308 >>>> auto.leader.rebalance.enable = true >>>> leader.imbalance.check.interval.seconds = 300 >>>> log.cleaner.min.cleanable.ratio = 0.5 >>>> replica.lag.time.max.ms = 10000 >>>> num.network.threads = 3 >>>> ssl.key.password = null >>>> reserved.broker.max.id = 1000 >>>> metrics.num.samples = 2 >>>> socket.send.buffer.bytes = 102400 >>>> ssl.protocol = TLS >>>> socket.receive.buffer.bytes = 102400 >>>> ssl.keystore.location = null >>>> replica.fetch.min.bytes = 1 >>>> broker.rack = null >>>> unclean.leader.election.enable = true >>>> sasl.enabled.mechanisms = [GSSAPI] >>>> group.min.session.timeout.ms = 6000 >>>> log.cleaner.io.buffer.load.factor = 0.9 >>>> offsets.retention.check.interval.ms = 600000 >>>> producer.purgatory.purge.interval.requests = 10000 >>>> metrics.sample.window.ms = 30000 >>>> broker.id = -1 >>>> offsets.topic.compression.codec = 0 >>>> log.retention.check.interval.ms = 300000 >>>> advertised.listeners = null >>>> leader.imbalance.per.broker.percentage = 10 >>>> (kafka.server.KafkaConfig) >>>> [2017-05-17 17:03:14,990] INFO starting (kafka.server.KafkaServer) >>>> [2017-05-17 17:03:14,998] INFO Connecting to zookeeper on >>>> ec2-34-223-200-113.us-west-2.compute.amazonaws.com:2181 >>>> (kafka.server.KafkaServer) >>>> [2017-05-17 17:03:40,740] FATAL Fatal error during KafkaServer startup. >>>> Prepare to shutdown (kafka.server.KafkaServer) >>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to >>>> zookeeper server within timeout: 25000 >>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232) >>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156) >>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130) >>>> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils. >>>> scala:75) >>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57) >>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294) >>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180) >>>> at kafka.server.KafkaServerStartable.startup(KafkaServerStartab >>>> le.scala:37) >>>> at kafka.Kafka$.main(Kafka.scala:67) >>>> at kafka.Kafka.main(Kafka.scala) >>>> [2017-05-17 17:03:40,740] INFO shutting down (kafka.server.KafkaServer) >>>> [2017-05-17 17:03:40,747] INFO shut down completed >>>> (kafka.server.KafkaServer) >>>> [2017-05-17 17:03:40,748] FATAL Fatal error during KafkaServerStartable >>>> startup. Prepare to shutdown (kafka.server.KafkaServerStartable) >>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to >>>> zookeeper server within timeout: 25000 >>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232) >>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156) >>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130) >>>> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils. >>>> scala:75) >>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57) >>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294) >>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180) >>>> at kafka.server.KafkaServerStartable.startup(KafkaServerStartab >>>> le.scala:37) >>>> at kafka.Kafka$.main(Kafka.scala:67) >>>> at kafka.Kafka.main(Kafka.scala) >>>> [2017-05-17 17:03:40,764] INFO shutting down (kafka.server.KafkaServer) >>>> >>>> Manually starting Kafka via Ambari fixes this... However, Metron REST >>>> still doesn't work: >>>> >>>> . ____ _ __ _ _ >>>> /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ >>>> ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ >>>> \\/ ___)| |_)| | | | | || (_| | ) ) ) ) >>>> ' |____| .__|_| |_|_| |_\__, | / / / / >>>> =========|_|==============|___/=/_/_/_/ >>>> :: Spring Boot :: (v1.4.1.RELEASE) >>>> >>>> 2017-05-17T17:03:52.406 ERROR [org.springframework.boot.Spri >>>> ngApplication] >>>> - Application startup failed >>>> org.springframework.beans.factory.UnsatisfiedDependencyException: Error >>>> creating bean with name 'kafkaServiceImpl' defined in URL >>>> [jar:file:/usr/metron/0.4.0/lib/metron-rest-0.4.0.jar!/BOOT- >>>> INF/classes!/org/apache/metron/rest/service/impl/KafkaServic >>>> eImpl.class]: >>>> Unsatisfied dependency expressed through constructor parameter 0; nested >>>> exception is org.springframework.beans.fact >>>> ory.UnsatisfiedDependencyException: >>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency >>>> expressed through field 'zkClient'; nested exception is >>>> org.springframework.beans.factory.BeanCreationException: Error creating >>>> bean with name 'zkClient' defined in class path resource >>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean >>>> instantiation >>>> via factory method failed; nested exception is >>>> org.springframework.beans.BeanInstantiationException: Failed to >>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' >>>> threw exception; nested exception is org.I0Itec.zkclient.exception. >>>> ZkTimeoutException: >>>> Unable to connect to zookeeper server within timeout: 10000 >>>> at org.springframework.beans.factory.support.ConstructorResolve >>>> r.createArgumentArray(ConstructorResolver.java:749) >>>> at org.springframework.beans.factory.support.ConstructorResolve >>>> r.autowireConstructor(ConstructorResolver.java:189) >>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>> pableBeanFactory.autowireConstructor(AbstractAu >>>> towireCapableBeanFactory.java:1148) >>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>> pableBeanFactory.createBeanInstance(AbstractAut >>>> owireCapableBeanFactory.java:1051) >>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac >>>> tory.java:510) >>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto >>>> ry.java:482) >>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>> y$1.getObject(AbstractBeanFactory.java:306) >>>> at org.springframework.beans.factory.support.DefaultSingletonBe >>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>> y.doGetBean(AbstractBeanFactory.java:302) >>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>> y.getBean(AbstractBeanFactory.java:197) >>>> at org.springframework.beans.factory.support.DefaultListableBea >>>> nFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751) >>>> at org.springframework.context.support.AbstractApplicationConte >>>> xt.finishBeanFactoryInitialization(AbstractApplicationContext.java:861) >>>> at org.springframework.context.support.AbstractApplicationConte >>>> xt.refresh(AbstractApplicationContext.java:541) >>>> at org.springframework.boot.context.embedded.EmbeddedWebApplica >>>> tionContext.refresh(EmbeddedWebApplicationContext.java:122) >>>> at org.springframework.boot.SpringApplication.refresh(SpringApp >>>> lication.java:761) >>>> at org.springframework.boot.SpringApplication.refreshContext(Sp >>>> ringApplication.java:371) >>>> at org.springframework.boot.SpringApplication.run(SpringApplica >>>> tion.java:315) >>>> at org.springframework.boot.SpringApplication.run(SpringApplica >>>> tion.java:1186) >>>> at org.springframework.boot.SpringApplication.run(SpringApplica >>>> tion.java:1175) >>>> at org.apache.metron.rest.MetronRestApplication.main(MetronRest >>>> Application.java:27) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >>>> ssorImpl.java:62) >>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>>> thodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>> at org.springframework.boot.loader.MainMethodRunner.run(MainMet >>>> hodRunner.java:48) >>>> at org.springframework.boot.loader.Launcher.launch(Launcher. >>>> java:87) >>>> at org.springframework.boot.loader.Launcher.launch(Launcher. >>>> java:50) >>>> at org.springframework.boot.loader.PropertiesLauncher.main( >>>> PropertiesLauncher.java:525) >>>> Caused by: org.springframework.beans.factory.UnsatisfiedDependencyExcep >>>> tion: >>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency >>>> expressed through field 'zkClient'; nested exception is >>>> org.springframework.beans.factory.BeanCreationException: Error creating >>>> bean with name 'zkClient' defined in class path resource >>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean >>>> instantiation >>>> via factory method failed; nested exception is >>>> org.springframework.beans.BeanInstantiationException: Failed to >>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' >>>> threw exception; nested exception is org.I0Itec.zkclient.exception. >>>> ZkTimeoutException: >>>> Unable to connect to zookeeper server within timeout: 10000 >>>> at org.springframework.beans.factory.annotation.AutowiredAnnota >>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A >>>> utowiredAnnotationBeanPostProcessor.java:569) >>>> at org.springframework.beans.factory.annotation.InjectionMetada >>>> ta.inject(InjectionMetadata.java:88) >>>> at org.springframework.beans.factory.annotation.AutowiredAnnota >>>> tionBeanPostProcessor.postProcessPropertyValues(AutowiredAnn >>>> otationBeanPostProcessor.java:349) >>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>> pableBeanFactory.populateBean(AbstractAutowireCapableBeanFac >>>> tory.java:1219) >>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac >>>> tory.java:543) >>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto >>>> ry.java:482) >>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>> y$1.getObject(AbstractBeanFactory.java:306) >>>> at org.springframework.beans.factory.support.DefaultSingletonBe >>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>> y.doGetBean(AbstractBeanFactory.java:302) >>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>> y.getBean(AbstractBeanFactory.java:197) >>>> at org.springframework.beans.factory.support.ConstructorResolve >>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:372) >>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow >>>> ireCapableBeanFactory.java:1128) >>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>> pableBeanFactory.createBeanInstance(AbstractAut >>>> owireCapableBeanFactory.java:1023) >>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac >>>> tory.java:510) >>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto >>>> ry.java:482) >>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>> y$1.getObject(AbstractBeanFactory.java:306) >>>> at org.springframework.beans.factory.support.DefaultSingletonBe >>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>> y.doGetBean(AbstractBeanFactory.java:302) >>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>> y.getBean(AbstractBeanFactory.java:202) >>>> at org.springframework.beans.factory.config.DependencyDescripto >>>> r.resolveCandidate(DependencyDescriptor.java:207) >>>> at org.springframework.beans.factory.support.DefaultListableBea >>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128) >>>> at org.springframework.beans.factory.support.DefaultListableBea >>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056) >>>> at org.springframework.beans.factory.support.ConstructorResolve >>>> r.resolveAutowiredArgument(ConstructorResolver.java:835) >>>> at org.springframework.beans.factory.support.ConstructorResolve >>>> r.createArgumentArray(ConstructorResolver.java:741) >>>> ... 27 more >>>> Caused by: org.springframework.beans.factory.BeanCreationException: >>>> Error >>>> creating bean with name 'zkClient' defined in class path resource >>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean >>>> instantiation >>>> via factory method failed; nested exception is >>>> org.springframework.beans.BeanInstantiationException: Failed to >>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' >>>> threw exception; nested exception is org.I0Itec.zkclient.exception. >>>> ZkTimeoutException: >>>> Unable to connect to zookeeper server within timeout: 10000 >>>> at org.springframework.beans.factory.support.ConstructorResolve >>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:599) >>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow >>>> ireCapableBeanFactory.java:1128) >>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>> pableBeanFactory.createBeanInstance(AbstractAut >>>> owireCapableBeanFactory.java:1023) >>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac >>>> tory.java:510) >>>> at org.springframework.beans.factory.support.AbstractAutowireCa >>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto >>>> ry.java:482) >>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>> y$1.getObject(AbstractBeanFactory.java:306) >>>> at org.springframework.beans.factory.support.DefaultSingletonBe >>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>> y.doGetBean(AbstractBeanFactory.java:302) >>>> at org.springframework.beans.factory.support.AbstractBeanFactor >>>> y.getBean(AbstractBeanFactory.java:202) >>>> at org.springframework.beans.factory.config.DependencyDescripto >>>> r.resolveCandidate(DependencyDescriptor.java:207) >>>> at org.springframework.beans.factory.support.DefaultListableBea >>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128) >>>> at org.springframework.beans.factory.support.DefaultListableBea >>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056) >>>> at org.springframework.beans.factory.annotation.AutowiredAnnota >>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A >>>> utowiredAnnotationBeanPostProcessor.java:566) >>>> ... 50 more >>>> Caused by: org.springframework.beans.BeanInstantiationException: Failed >>>> to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method >>>> 'zkClient' >>>> threw exception; nested exception is org.I0Itec.zkclient.exception. >>>> ZkTimeoutException: >>>> Unable to connect to zookeeper server within timeout: 10000 >>>> at org.springframework.beans.factory.support.SimpleInstantiatio >>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:189) >>>> at org.springframework.beans.factory.support.ConstructorResolve >>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:588) >>>> ... 62 more >>>> Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to >>>> connect to zookeeper server within timeout: 10000 >>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232) >>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156) >>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130) >>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:97) >>>> at org.apache.metron.rest.config.ZookeeperConfig.zkClient(Zooke >>>> eperConfig.java:52) >>>> at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr >>>> ingCGLIB$$6e0c6742.CGLIB$zkClient$1(<generated>) >>>> at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr >>>> ingCGLIB$$6e0c6742$$FastClassBySpringCGLIB$$3219d861.invoke( >>>> <generated>) >>>> at org.springframework.cglib.proxy.MethodProxy.invokeSuper(Meth >>>> odProxy.java:228) >>>> at org.springframework.context.annotation.ConfigurationClassEnh >>>> ancer$BeanMethodInterceptor.intercept(ConfigurationClassEnha >>>> ncer.java:356) >>>> at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr >>>> ingCGLIB$$6e0c6742.zkClient(<generated>) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >>>> ssorImpl.java:62) >>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>>> thodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>> at org.springframework.beans.factory.support.SimpleInstantiatio >>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:162) >>>> ... 63 more >>>> >>>> Any ideas on what might be going or why the timeout triggers? >>>> >>>> >>>> On 2017-05-17 08:31, Laurens Vets wrote: >>>> >>>> Hello list, >>>>> >>>>> I'm trying to spin up Metron in EC2 for testing purposes. I made the >>>>> following changes in playbook.yml so that I'm left with 5 EC2 >>>>> instances instead of 10: >>>>> >>>>> <-- SNIP --> >>>>> tasks: >>>>> - include: tasks/create-keypair.yml >>>>> - include: tasks/create-vpc.yml >>>>> - include: tasks/create-open-inbound-security-group.yml >>>>> - include: tasks/create-open-outbound-security-group.yml >>>>> - include: tasks/create-hosts.yml host_count=1 >>>>> host_type=sensors,ambari_master,ec2,monit >>>>> # - include: tasks/create-hosts.yml host_count=4 >>>>> host_type=ambari_slave,ec2 >>>>> - include: tasks/create-hosts.yml host_count=1 >>>>> host_type=pcap_server,monit,ec2 >>>>> - include: tasks/create-hosts.yml host_count=1 >>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin >>>>> # - include: tasks/create-hosts.yml host_count=2 >>>>> host_type=ambari_slave,search,ec2 >>>>> - include: tasks/create-hosts.yml host_count=1 >>>>> host_type=ambari_slave,search,ec2 >>>>> - include: tasks/create-hosts.yml host_count=1 >>>>> host_type=ambari_slave,web,ec2 >>>>> <-- END SNIP --> >>>>> >>>>> I also had to change the line "ansible-playbook -vvv -i ec2.py >>>>> playbook.yml \ >>>>> --skip-tags="quick_dev,sensor-stubs" \ >>>>> --extra-vars="env=$ENV" \ >>>>> $EXTRA_ARGS" >>>>> >>>>> to "ansible-playbook -vvv -i ec2.py playbook.yml \ >>>>> --skip-tags="quick_dev,sensor-stubs" \ >>>>> --extra-vars="env=$ENV" \ >>>>> $EXTRA_ARGS -c paramiko" >>>>> in run.sh otherwise ssh would time out. >>>>> >>>>> However, Ansible errors out with: >>>>> >>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | TASK [ambari_config : >>>>> Deploy cluster with Ambari; >>>>> http://ec2-35-164-26-255.us-west-2.compute.amazonaws.com:8080] *** >>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | task path: >>>>> /home/laurens/SAPSource/metron/metron-deployment/roles/ >>>>> ambari_config/tasks/main.yml:36 >>>>> 2017-05-16 18:02:52,660 p=5362 u=laurens | fatal: >>>>> [ec2-35-164-26-255.us-west-2.compute.amazonaws.com]: FAILED! => >>>>> {"changed": false, "failed": true, "invocation": {"module_args": >>>>> {"blueprint_name": "metron_blueprint", "blueprint_var": {"groups": >>>>> [{"cardinality": 1, "components": [{"name": "NAMENODE"}, {"name": >>>>> "SECONDARY_NAMENODE"}, {"name": "RESOURCEMANAGER"}, {"name": >>>>> "HISTORYSERVER"}, {"name": "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, >>>>> {"name": "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name": >>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name": "ES_SLAVE"}], >>>>> "configuration": [], "name": "master_1"}, {"cardinality": 1, >>>>> "components": [{"name": "ZOOKEEPER_SERVER"}, {"name": "NIMBUS"}, >>>>> {"name": "STORM_UI_SERVER"}, {"name": "DRPC_SERVER"}, {"name": >>>>> "SPARK_JOBHISTORYSERVER"}, {"name": "HBASE_MASTER"}, {"name": >>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name": >>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name": >>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name": >>>>> "APP_TIMELINE_SERVER"}, {"name": "ES_SLAVE"}], "configuration": [], >>>>> "name": "master_2"}, {"cardinality": 1, "components": [{"name": >>>>> "METRON_INDEXING"}, {"name": "METRON_ENRICHMENT_MASTER"}, {"name": >>>>> "METRON_PARSERS"}, {"name": "METRON_REST"}, {"name": "DATANODE"}, >>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name": >>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name": >>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name": >>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name": >>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [], >>>>> "name": "metron"}, {"cardinality": 1, "components": [{"name": >>>>> "KIBANA_MASTER"}, {"name": "ES_MASTER"}], "configuration": [], "name": >>>>> "web"}, {"cardinality": "1+", "components": [{"name": "DATANODE"}, >>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name": >>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name": >>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name": >>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name": >>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [], >>>>> "name": "slaves"}], "required_configurations": [{"metron-env": >>>>> {"es_hosts": >>>>> "ec2-35-164-136-88.us-west-2.compute.amazonaws.com,ec2-34-20 >>>>> 8-94-45.us-west-2.compute.amazonaws.com", >>>>> "metron_jdbc_driver": "org.h2.Driver", "metron_jdbc_password": "root", >>>>> "metron_jdbc_platform": "h2", "metron_jdbc_url": >>>>> "jdbc:h2:file:~/metrondb", "metron_jdbc_username": "root", >>>>> "storm_rest_addr": >>>>> "ec2-34-208-94-45.us-west-2.compute.amazonaws.com:8744", >>>>> "zeppelin_server_url": >>>>> "ec2-34-210-151-29.us-west-2.compute.amazonaws.com"}}, {"kibana-env": >>>>> {"kibana_default_application": "dashboard/Metron-Dashboard", >>>>> "kibana_es_url": >>>>> "http://ec2-35-164-136-88.us-west-2.compute.amazonaws.com:9200";, >>>>> "kibana_log_dir": "/var/log/kibana", "kibana_pid_dir": >>>>> "/var/run/kibana", "kibana_server_port": 5000}}], "stack_name": "HDP", >>>>> "stack_version": "2.5"}, "cluster_name": "metron", "cluster_state": >>>>> "present", "configurations": [{"zoo.cfg": {"dataDir": >>>>> "/data1/hadoop/zookeeper"}}, {"hadoop-env": {"dtnode_heapsize": 1024, >>>>> "namenode_heapsize": 2048}}, {"hbase-env": {"hbase_master_heapsize": >>>>> 1024, "hbase_regionserver_heapsize": 1024}}, {"hdfs-site": >>>>> {"dfs.datanode.data.dir": >>>>> "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data", >>>>> "dfs.journalnode.edits.dir": "/data1/hadoop/hdfs/journalnode", >>>>> "dfs.namenode.checkpoint.dir": "/data1/hadoop/hdfs/namesecondary", >>>>> "dfs.namenode.name.dir": "/data1/hadoop/hdfs/namenode"}}, >>>>> {"mapred-site": {"mapreduce.jobhistory.recovery.store.leveldb.path": >>>>> "/data1/hadoop/mapreduce/jhs", "mapreduce.map.memory.mb": 1229, >>>>> "mapreduce.reduce.memory.mb": 1229}}, {"yarn-site": >>>>> {"yarn.nodemanager.local-dirs": "/data1/hadoop/yarn/local", >>>>> "yarn.nodemanager.log-dirs": "/data1/hadoop/yarn/log", >>>>> "yarn.nodemanager.resource.memory-mb": 4096, >>>>> "yarn.timeline-service.leveldb-state-store.path": >>>>> "/data1/hadoop/yarn/timeline", >>>>> "yarn.timeline-service.leveldb-timeline-store.path": >>>>> "/data1/hadoop/yarn/timeline"}}, {"storm-site": {"storm.local.dir": >>>>> "/data1/hadoop/storm", "supervisor.slots.ports": "[6700, 6701, 6702, >>>>> 6703]", "topology.classpath": "/etc/hbase/conf:/etc/hadoop/conf"}}, >>>>> {"kafka-broker": {"log.dirs": "/data2/kafka-log"}}, {"metron-env": >>>>> {"parsers": "bro,snort,yaf"}}, {"elastic-site": >>>>> {"gateway_recover_after_data_nodes": 1, "index_number_of_replicas": 1, >>>>> "index_number_of_shards": 2, "network_host": "[ _local_, _eth0_ ]", >>>>> "zen_discovery_ping_unicast_hosts": "[ >>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com, >>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com ]"}}], "host": >>>>> "ec2-35-164-26-255.us-west-2.compute.amazonaws.com", "password": >>>>> "admin", "port": 8080, "username": "admin", "wait_for_complete": >>>>> true}, "module_name": "ambari_cluster_state"}, "msg": "Ambari client >>>>> exception occurred: Could not create cluster: request code 400, >>>>> request message {\n \"status\" : 400,\n \"message\" : >>>>> \"Invalid Cluster Creation Template: >>>>> org.apache.ambari.server.topology.InvalidTopologyTemplateException: >>>>> Host group 'slaves' must contain at least one 'hosts/fqdn' or a >>>>> 'host_count' value\"\n}"} >>>>> >>>>> PLAY RECAP ****************************** >>>>> ****************************** >>>>> ********* >>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com : ok=49 changed=27 >>>>> unreachable=0 failed=0 >>>>> ec2-34-209-224-190.us-west-2.compute.amazonaws.com : ok=23 changed=8 >>>>> unreachable=0 failed=0 >>>>> ec2-34-210-151-29.us-west-2.compute.amazonaws.com : ok=58 changed=31 >>>>> unreachable=0 failed=0 >>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com : ok=49 changed=27 >>>>> unreachable=0 failed=0 >>>>> ec2-35-164-26-255.us-west-2.compute.amazonaws.com : ok=54 changed=28 >>>>> unreachable=0 failed=1 >>>>> localhost : ok=29 changed=12 unreachable=0 >>>>> failed=0 >>>>> >>>>> Any idea what might be going on? >>>>> >>>>> (Don't worry about the URLs, the instances have been terminated) >>>>> >>>>> >>>>
