Missed one important point. You actually choose the data center here [2] in the configuration file.
[2] https://github.com/apache/metron/blob/master/metron-deployment/amazon-ec2/conf/defaults.yml#L43 On Fri, May 19, 2017 at 1:52 PM, Nick Allen <n...@nickallen.org> wrote: > I am not sure what the issue is. Nothing is jumping out at me. > > Weird issues can arise when the machine that you are running the Ansible > deployment from has intermittent connectivity issues to the AWS data center. > > Is Amazon's us-west-2 data center the one closest to you? You might try a > different/closer data center. You can alter the data center here [1]. > > [1] https://github.com/apache/metron/blob/master/metron- > deployment/amazon-ec2/conf/defaults.yml#L27-L39 > > Other than that, you could try standing up a small node in AWS and then > use that to run the Metron deployment from. That always works much more > smoothly. > > > > > On Wed, May 17, 2017 at 4:41 PM, Laurens Vets <laur...@daemon.be> wrote: > >> Also, it's only 1 out of 10 hosts that failing it seems. >> >> >> On 2017-05-17 13:25, Laurens Vets wrote: >> >>> When you say Metron host, do you mean the host I'm pushing from to ec2 >>> or the master ambari server in ec2? >>> >>> On 2017-05-17 11:51, Ryan Merriman wrote: >>> >>>> That happens when you don't have the zookeeper url configured correctly. >>>> Can you check the contents of the /etc/sysconfig/metron file on the >>>> Metron >>>> host? >>>> >>>> On Wed, May 17, 2017 at 1:36 PM, Laurens Vets <laur...@daemon.be> >>>> wrote: >>>> >>>> For testing purposes, I decided to spin up the default Metron AWS >>>>> config. >>>>> This resulted in a hang from ansible here: >>>>> >>>>> TASK [librdkafka : include] ****************************** >>>>> ********************** >>>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/ >>>>> librdkafka/tasks/main.yml:18 >>>>> included: /home/laurens/SAPSource/metron/metron-deployment/roles/ >>>>> librdkafka/tasks/dependencies.yml for ec2-34-210-194-189.us-west-2.c >>>>> ompute.amazonaws.com >>>>> >>>>> TASK [librdkafka : Install prerequisites] >>>>> ****************************** >>>>> ******** >>>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/ >>>>> librdkafka/tasks/dependencies.yml:18 >>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> ESTABLISH >>>>> CONNECTION >>>>> FOR USER: centos on PORT 22 TO ec2-34-210-194-189.us-west-2.c >>>>> ompute.amazonaws.com >>>>> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141: >>>>> FutureWarning: CTR mode needs counter parameter, not IV >>>>> self._cipher = factory.new(key, *args, **kwargs) >>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC ( umask 22 >>>>> && >>>>> mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp >>>>> -1495041091.74-92163853889508 >>>>> )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp >>>>> -1495041091.74-92163853889508 >>>>> )" ) >>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> PUT >>>>> /tmp/tmpwnH61y >>>>> TO /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853 >>>>> 889508/yum >>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC /bin/sh -c >>>>> 'sudo -H -S -n -u root /bin/sh -c '"'"'echo >>>>> BECOME-SUCCESS-rmswjjyhfdywqvwtvqwcmbsqpsbohvxh; >>>>> LANG=en_CA.UTF-8 LC_ALL=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8 >>>>> /usr/bin/python >>>>> -tt /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853 >>>>> 889508/yum; >>>>> rm -rf "/home/centos/.ansible/tmp/ansible-tmp-1495041091.74-9216385 >>>>> 3889508/" >>>>> > /dev/null 2>&1'"'"'' >>>>> >>>>> Looking in the machine logs, I see the following for Kafka and Metron >>>>> REST: >>>>> >>>>> Kafka: >>>>> [2017-05-17 17:03:14,831] INFO KafkaConfig values: >>>>> advertised.host.name = null >>>>> metric.reporters = [] >>>>> quota.producer.default = 9223372036854775807 >>>>> offsets.topic.num.partitions = 50 >>>>> log.flush.interval.messages = 9223372036854775807 >>>>> auto.create.topics.enable = true >>>>> controller.socket.timeout.ms = 30000 >>>>> log.flush.interval.ms = null >>>>> principal.builder.class = class org.apache.kafka.common.securi >>>>> ty.auth.DefaultPrincipalBuilder >>>>> replica.socket.receive.buffer.bytes = 65536 >>>>> min.insync.replicas = 1 >>>>> replica.fetch.wait.max.ms = 500 >>>>> num.recovery.threads.per.data.dir = 1 >>>>> ssl.keystore.type = JKS >>>>> sasl.mechanism.inter.broker.protocol = GSSAPI >>>>> default.replication.factor = 1 >>>>> ssl.truststore.password = null >>>>> log.preallocate = false >>>>> sasl.kerberos.principal.to.local.rules = [DEFAULT] >>>>> fetch.purgatory.purge.interval.requests = 10000 >>>>> ssl.endpoint.identification.algorithm = null >>>>> replica.socket.timeout.ms = 30000 >>>>> message.max.bytes = 1000000 >>>>> num.io.threads = 8 >>>>> offsets.commit.required.acks = -1 >>>>> log.flush.offset.checkpoint.interval.ms = 60000 >>>>> delete.topic.enable = false >>>>> quota.window.size.seconds = 1 >>>>> ssl.truststore.type = JKS >>>>> offsets.commit.timeout.ms = 5000 >>>>> quota.window.num = 11 >>>>> zookeeper.connect = ec2-34-223-200-113.us-west-2.c >>>>> ompute.amazonaws.com:2181 >>>>> authorizer.class.name = >>>>> num.replica.fetchers = 1 >>>>> log.retention.ms = null >>>>> log.roll.jitter.hours = 0 >>>>> log.cleaner.enable = true >>>>> offsets.load.buffer.size = 5242880 >>>>> log.cleaner.delete.retention.ms = 86400000 >>>>> ssl.client.auth = none >>>>> controlled.shutdown.max.retries = 3 >>>>> queued.max.requests = 500 >>>>> offsets.topic.replication.factor = 3 >>>>> log.cleaner.threads = 1 >>>>> sasl.kerberos.service.name = null >>>>> sasl.kerberos.ticket.renew.jitter = 0.05 >>>>> socket.request.max.bytes = 104857600 >>>>> ssl.trustmanager.algorithm = PKIX >>>>> zookeeper.session.timeout.ms = 30000 >>>>> log.retention.bytes = -1 >>>>> log.message.timestamp.type = CreateTime >>>>> sasl.kerberos.min.time.before.relogin = 60000 >>>>> zookeeper.set.acl = false >>>>> connections.max.idle.ms = 600000 >>>>> offsets.retention.minutes = 86400000 >>>>> replica.fetch.backoff.ms = 1000 >>>>> inter.broker.protocol.version = 0.10.0-IV1 >>>>> log.retention.hours = 168 >>>>> num.partitions = 1 >>>>> broker.id.generation.enable = true >>>>> listeners = PLAINTEXT://ec2-34-209-53-166. >>>>> us-west-2.compute.amazonaws.com:6667 >>>>> ssl.provider = null >>>>> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] >>>>> log.roll.ms = null >>>>> log.flush.scheduler.interval.ms = 9223372036854775807 >>>>> ssl.cipher.suites = null >>>>> log.index.size.max.bytes = 10485760 >>>>> ssl.keymanager.algorithm = SunX509 >>>>> security.inter.broker.protocol = PLAINTEXT >>>>> replica.fetch.max.bytes = 1048576 >>>>> advertised.port = null >>>>> log.cleaner.dedupe.buffer.size = 134217728 >>>>> replica.high.watermark.checkpoint.interval.ms = 5000 >>>>> log.cleaner.io.buffer.size = 524288 >>>>> sasl.kerberos.ticket.renew.window.factor = 0.8 >>>>> zookeeper.connection.timeout.ms = 25000 >>>>> controlled.shutdown.retry.backoff.ms = 5000 >>>>> log.roll.hours = 168 >>>>> log.cleanup.policy = delete >>>>> host.name = >>>>> log.roll.jitter.ms = null >>>>> max.connections.per.ip = 2147483647 >>>>> offsets.topic.segment.bytes = 104857600 >>>>> background.threads = 10 >>>>> quota.consumer.default = 9223372036854775807 >>>>> request.timeout.ms = 30000 >>>>> log.message.format.version = 0.10.0-IV1 >>>>> log.index.interval.bytes = 4096 >>>>> log.dir = /tmp/kafka-logs >>>>> log.segment.bytes = 1073741824 >>>>> log.cleaner.backoff.ms = 15000 >>>>> offset.metadata.max.bytes = 4096 >>>>> ssl.truststore.location = null >>>>> group.max.session.timeout.ms = 300000 >>>>> ssl.keystore.password = null >>>>> zookeeper.sync.time.ms = 2000 >>>>> port = 6667 >>>>> log.retention.minutes = null >>>>> log.segment.delete.delay.ms = 60000 >>>>> log.dirs = /data2/kafka-log >>>>> controlled.shutdown.enable = true >>>>> compression.type = producer >>>>> max.connections.per.ip.overrides = >>>>> log.message.timestamp.difference.max.ms = 9223372036854775807 >>>>> sasl.kerberos.kinit.cmd = /usr/bin/kinit >>>>> log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308 >>>>> auto.leader.rebalance.enable = true >>>>> leader.imbalance.check.interval.seconds = 300 >>>>> log.cleaner.min.cleanable.ratio = 0.5 >>>>> replica.lag.time.max.ms = 10000 >>>>> num.network.threads = 3 >>>>> ssl.key.password = null >>>>> reserved.broker.max.id = 1000 >>>>> metrics.num.samples = 2 >>>>> socket.send.buffer.bytes = 102400 >>>>> ssl.protocol = TLS >>>>> socket.receive.buffer.bytes = 102400 >>>>> ssl.keystore.location = null >>>>> replica.fetch.min.bytes = 1 >>>>> broker.rack = null >>>>> unclean.leader.election.enable = true >>>>> sasl.enabled.mechanisms = [GSSAPI] >>>>> group.min.session.timeout.ms = 6000 >>>>> log.cleaner.io.buffer.load.factor = 0.9 >>>>> offsets.retention.check.interval.ms = 600000 >>>>> producer.purgatory.purge.interval.requests = 10000 >>>>> metrics.sample.window.ms = 30000 >>>>> broker.id = -1 >>>>> offsets.topic.compression.codec = 0 >>>>> log.retention.check.interval.ms = 300000 >>>>> advertised.listeners = null >>>>> leader.imbalance.per.broker.percentage = 10 >>>>> (kafka.server.KafkaConfig) >>>>> [2017-05-17 17:03:14,990] INFO starting (kafka.server.KafkaServer) >>>>> [2017-05-17 17:03:14,998] INFO Connecting to zookeeper on >>>>> ec2-34-223-200-113.us-west-2.compute.amazonaws.com:2181 >>>>> (kafka.server.KafkaServer) >>>>> [2017-05-17 17:03:40,740] FATAL Fatal error during KafkaServer startup. >>>>> Prepare to shutdown (kafka.server.KafkaServer) >>>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to >>>>> zookeeper server within timeout: 25000 >>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232) >>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156) >>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130) >>>>> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils. >>>>> scala:75) >>>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57) >>>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294) >>>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180) >>>>> at kafka.server.KafkaServerStarta >>>>> ble.startup(KafkaServerStartab >>>>> le.scala:37) >>>>> at kafka.Kafka$.main(Kafka.scala:67) >>>>> at kafka.Kafka.main(Kafka.scala) >>>>> [2017-05-17 17:03:40,740] INFO shutting down (kafka.server.KafkaServer) >>>>> [2017-05-17 17:03:40,747] INFO shut down completed >>>>> (kafka.server.KafkaServer) >>>>> [2017-05-17 17:03:40,748] FATAL Fatal error during KafkaServerStartable >>>>> startup. Prepare to shutdown (kafka.server.KafkaServerStartable) >>>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to >>>>> zookeeper server within timeout: 25000 >>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232) >>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156) >>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130) >>>>> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils. >>>>> scala:75) >>>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57) >>>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294) >>>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180) >>>>> at kafka.server.KafkaServerStarta >>>>> ble.startup(KafkaServerStartab >>>>> le.scala:37) >>>>> at kafka.Kafka$.main(Kafka.scala:67) >>>>> at kafka.Kafka.main(Kafka.scala) >>>>> [2017-05-17 17:03:40,764] INFO shutting down (kafka.server.KafkaServer) >>>>> >>>>> Manually starting Kafka via Ambari fixes this... However, Metron REST >>>>> still doesn't work: >>>>> >>>>> . ____ _ __ _ _ >>>>> /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ >>>>> ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ >>>>> \\/ ___)| |_)| | | | | || (_| | ) ) ) ) >>>>> ' |____| .__|_| |_|_| |_\__, | / / / / >>>>> =========|_|==============|___/=/_/_/_/ >>>>> :: Spring Boot :: (v1.4.1.RELEASE) >>>>> >>>>> 2017-05-17T17:03:52.406 ERROR [org.springframework.boot.Spri >>>>> ngApplication] >>>>> - Application startup failed >>>>> org.springframework.beans.factory.UnsatisfiedDependencyException: >>>>> Error >>>>> creating bean with name 'kafkaServiceImpl' defined in URL >>>>> [jar:file:/usr/metron/0.4.0/lib/metron-rest-0.4.0.jar!/BOOT- >>>>> INF/classes!/org/apache/metron/rest/service/impl/KafkaServic >>>>> eImpl.class]: >>>>> Unsatisfied dependency expressed through constructor parameter 0; >>>>> nested >>>>> exception is org.springframework.beans.fact >>>>> ory.UnsatisfiedDependencyException: >>>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency >>>>> expressed through field 'zkClient'; nested exception is >>>>> org.springframework.beans.factory.BeanCreationException: Error >>>>> creating >>>>> bean with name 'zkClient' defined in class path resource >>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean >>>>> instantiation >>>>> via factory method failed; nested exception is >>>>> org.springframework.beans.BeanInstantiationException: Failed to >>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' >>>>> threw exception; nested exception is org.I0Itec.zkclient.exception. >>>>> ZkTimeoutException: >>>>> Unable to connect to zookeeper server within timeout: 10000 >>>>> at org.springframework.beans.fact >>>>> ory.support.ConstructorResolve >>>>> r.createArgumentArray(ConstructorResolver.java:749) >>>>> at org.springframework.beans.fact >>>>> ory.support.ConstructorResolve >>>>> r.autowireConstructor(ConstructorResolver.java:189) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractAutowireCa >>>>> pableBeanFactory.autowireConstructor(AbstractAu >>>>> towireCapableBeanFactory.java:1148) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractAutowireCa >>>>> pableBeanFactory.createBeanInstance(AbstractAut >>>>> owireCapableBeanFactory.java:1051) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractAutowireCa >>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac >>>>> tory.java:510) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractAutowireCa >>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto >>>>> ry.java:482) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractBeanFactor >>>>> y$1.getObject(AbstractBeanFactory.java:306) >>>>> at org.springframework.beans.fact >>>>> ory.support.DefaultSingletonBe >>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractBeanFactor >>>>> y.doGetBean(AbstractBeanFactory.java:302) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractBeanFactor >>>>> y.getBean(AbstractBeanFactory.java:197) >>>>> at org.springframework.beans.fact >>>>> ory.support.DefaultListableBea >>>>> nFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751) >>>>> at org.springframework.context.su >>>>> pport.AbstractApplicationConte >>>>> xt.finishBeanFactoryInitialization(AbstractApplicationContex >>>>> t.java:861) >>>>> at org.springframework.context.su >>>>> pport.AbstractApplicationConte >>>>> xt.refresh(AbstractApplicationContext.java:541) >>>>> at org.springframework.boot.conte >>>>> xt.embedded.EmbeddedWebApplica >>>>> tionContext.refresh(EmbeddedWebApplicationContext.java:122) >>>>> at org.springframework.boot.Sprin >>>>> gApplication.refresh(SpringApp >>>>> lication.java:761) >>>>> at org.springframework.boot.Sprin >>>>> gApplication.refreshContext(Sp >>>>> ringApplication.java:371) >>>>> at org.springframework.boot.Sprin >>>>> gApplication.run(SpringApplica >>>>> tion.java:315) >>>>> at org.springframework.boot.Sprin >>>>> gApplication.run(SpringApplica >>>>> tion.java:1186) >>>>> at org.springframework.boot.Sprin >>>>> gApplication.run(SpringApplica >>>>> tion.java:1175) >>>>> at org.apache.metron.rest.MetronR >>>>> estApplication.main(MetronRest >>>>> Application.java:27) >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>> at sun.reflect.NativeMethodAccess >>>>> orImpl.invoke(NativeMethodAcce >>>>> ssorImpl.java:62) >>>>> at sun.reflect.DelegatingMethodAc >>>>> cessorImpl.invoke(DelegatingMe >>>>> thodAccessorImpl.java:43) >>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>> at org.springframework.boot.loade >>>>> r.MainMethodRunner.run(MainMet >>>>> hodRunner.java:48) >>>>> at org.springframework.boot.loader.Launcher.launch(Launcher. >>>>> java:87) >>>>> at org.springframework.boot.loader.Launcher.launch(Launcher. >>>>> java:50) >>>>> at org.springframework.boot.loader.PropertiesLauncher.main( >>>>> PropertiesLauncher.java:525) >>>>> Caused by: org.springframework.beans.fact >>>>> ory.UnsatisfiedDependencyException: >>>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency >>>>> expressed through field 'zkClient'; nested exception is >>>>> org.springframework.beans.factory.BeanCreationException: Error >>>>> creating >>>>> bean with name 'zkClient' defined in class path resource >>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean >>>>> instantiation >>>>> via factory method failed; nested exception is >>>>> org.springframework.beans.BeanInstantiationException: Failed to >>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' >>>>> threw exception; nested exception is org.I0Itec.zkclient.exception. >>>>> ZkTimeoutException: >>>>> Unable to connect to zookeeper server within timeout: 10000 >>>>> at org.springframework.beans.fact >>>>> ory.annotation.AutowiredAnnota >>>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A >>>>> utowiredAnnotationBeanPostProcessor.java:569) >>>>> at org.springframework.beans.fact >>>>> ory.annotation.InjectionMetada >>>>> ta.inject(InjectionMetadata.java:88) >>>>> at org.springframework.beans.fact >>>>> ory.annotation.AutowiredAnnota >>>>> tionBeanPostProcessor.postProcessPropertyValues(AutowiredAnn >>>>> otationBeanPostProcessor.java:349) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractAutowireCa >>>>> pableBeanFactory.populateBean(AbstractAutowireCapableBeanFac >>>>> tory.java:1219) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractAutowireCa >>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac >>>>> tory.java:543) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractAutowireCa >>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto >>>>> ry.java:482) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractBeanFactor >>>>> y$1.getObject(AbstractBeanFactory.java:306) >>>>> at org.springframework.beans.fact >>>>> ory.support.DefaultSingletonBe >>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractBeanFactor >>>>> y.doGetBean(AbstractBeanFactory.java:302) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractBeanFactor >>>>> y.getBean(AbstractBeanFactory.java:197) >>>>> at org.springframework.beans.fact >>>>> ory.support.ConstructorResolve >>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:372) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractAutowireCa >>>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow >>>>> ireCapableBeanFactory.java:1128) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractAutowireCa >>>>> pableBeanFactory.createBeanInstance(AbstractAut >>>>> owireCapableBeanFactory.java:1023) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractAutowireCa >>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac >>>>> tory.java:510) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractAutowireCa >>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto >>>>> ry.java:482) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractBeanFactor >>>>> y$1.getObject(AbstractBeanFactory.java:306) >>>>> at org.springframework.beans.fact >>>>> ory.support.DefaultSingletonBe >>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractBeanFactor >>>>> y.doGetBean(AbstractBeanFactory.java:302) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractBeanFactor >>>>> y.getBean(AbstractBeanFactory.java:202) >>>>> at org.springframework.beans.fact >>>>> ory.config.DependencyDescripto >>>>> r.resolveCandidate(DependencyDescriptor.java:207) >>>>> at org.springframework.beans.fact >>>>> ory.support.DefaultListableBea >>>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128) >>>>> at org.springframework.beans.fact >>>>> ory.support.DefaultListableBea >>>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056) >>>>> at org.springframework.beans.fact >>>>> ory.support.ConstructorResolve >>>>> r.resolveAutowiredArgument(ConstructorResolver.java:835) >>>>> at org.springframework.beans.fact >>>>> ory.support.ConstructorResolve >>>>> r.createArgumentArray(ConstructorResolver.java:741) >>>>> ... 27 more >>>>> Caused by: org.springframework.beans.factory.BeanCreationException: >>>>> Error >>>>> creating bean with name 'zkClient' defined in class path resource >>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean >>>>> instantiation >>>>> via factory method failed; nested exception is >>>>> org.springframework.beans.BeanInstantiationException: Failed to >>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient' >>>>> threw exception; nested exception is org.I0Itec.zkclient.exception. >>>>> ZkTimeoutException: >>>>> Unable to connect to zookeeper server within timeout: 10000 >>>>> at org.springframework.beans.fact >>>>> ory.support.ConstructorResolve >>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:599) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractAutowireCa >>>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow >>>>> ireCapableBeanFactory.java:1128) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractAutowireCa >>>>> pableBeanFactory.createBeanInstance(AbstractAut >>>>> owireCapableBeanFactory.java:1023) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractAutowireCa >>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac >>>>> tory.java:510) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractAutowireCa >>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto >>>>> ry.java:482) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractBeanFactor >>>>> y$1.getObject(AbstractBeanFactory.java:306) >>>>> at org.springframework.beans.fact >>>>> ory.support.DefaultSingletonBe >>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractBeanFactor >>>>> y.doGetBean(AbstractBeanFactory.java:302) >>>>> at org.springframework.beans.fact >>>>> ory.support.AbstractBeanFactor >>>>> y.getBean(AbstractBeanFactory.java:202) >>>>> at org.springframework.beans.fact >>>>> ory.config.DependencyDescripto >>>>> r.resolveCandidate(DependencyDescriptor.java:207) >>>>> at org.springframework.beans.fact >>>>> ory.support.DefaultListableBea >>>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128) >>>>> at org.springframework.beans.fact >>>>> ory.support.DefaultListableBea >>>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056) >>>>> at org.springframework.beans.fact >>>>> ory.annotation.AutowiredAnnota >>>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A >>>>> utowiredAnnotationBeanPostProcessor.java:566) >>>>> ... 50 more >>>>> Caused by: org.springframework.beans.BeanInstantiationException: >>>>> Failed >>>>> to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method >>>>> 'zkClient' >>>>> threw exception; nested exception is org.I0Itec.zkclient.exception. >>>>> ZkTimeoutException: >>>>> Unable to connect to zookeeper server within timeout: 10000 >>>>> at org.springframework.beans.fact >>>>> ory.support.SimpleInstantiatio >>>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:189) >>>>> at org.springframework.beans.fact >>>>> ory.support.ConstructorResolve >>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:588) >>>>> ... 62 more >>>>> Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to >>>>> connect to zookeeper server within timeout: 10000 >>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232) >>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156) >>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130) >>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:97) >>>>> at org.apache.metron.rest.config. >>>>> ZookeeperConfig.zkClient(Zooke >>>>> eperConfig.java:52) >>>>> at org.apache.metron.rest.config. >>>>> ZookeeperConfig$$EnhancerBySpr >>>>> ingCGLIB$$6e0c6742.CGLIB$zkClient$1(<generated>) >>>>> at org.apache.metron.rest.config. >>>>> ZookeeperConfig$$EnhancerBySpr >>>>> ingCGLIB$$6e0c6742$$FastClassBySpringCGLIB$$3219d861.invoke( >>>>> <generated>) >>>>> at org.springframework.cglib.prox >>>>> y.MethodProxy.invokeSuper(Meth >>>>> odProxy.java:228) >>>>> at org.springframework.context.an >>>>> notation.ConfigurationClassEnh >>>>> ancer$BeanMethodInterceptor.intercept(ConfigurationClassEnha >>>>> ncer.java:356) >>>>> at org.apache.metron.rest.config. >>>>> ZookeeperConfig$$EnhancerBySpr >>>>> ingCGLIB$$6e0c6742.zkClient(<generated>) >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>> at sun.reflect.NativeMethodAccess >>>>> orImpl.invoke(NativeMethodAcce >>>>> ssorImpl.java:62) >>>>> at sun.reflect.DelegatingMethodAc >>>>> cessorImpl.invoke(DelegatingMe >>>>> thodAccessorImpl.java:43) >>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>> at org.springframework.beans.fact >>>>> ory.support.SimpleInstantiatio >>>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:162) >>>>> ... 63 more >>>>> >>>>> Any ideas on what might be going or why the timeout triggers? >>>>> >>>>> >>>>> On 2017-05-17 08:31, Laurens Vets wrote: >>>>> >>>>> Hello list, >>>>>> >>>>>> I'm trying to spin up Metron in EC2 for testing purposes. I made the >>>>>> following changes in playbook.yml so that I'm left with 5 EC2 >>>>>> instances instead of 10: >>>>>> >>>>>> <-- SNIP --> >>>>>> tasks: >>>>>> - include: tasks/create-keypair.yml >>>>>> - include: tasks/create-vpc.yml >>>>>> - include: tasks/create-open-inbound-security-group.yml >>>>>> - include: tasks/create-open-outbound-security-group.yml >>>>>> - include: tasks/create-hosts.yml host_count=1 >>>>>> host_type=sensors,ambari_master,ec2,monit >>>>>> # - include: tasks/create-hosts.yml host_count=4 >>>>>> host_type=ambari_slave,ec2 >>>>>> - include: tasks/create-hosts.yml host_count=1 >>>>>> host_type=pcap_server,monit,ec2 >>>>>> - include: tasks/create-hosts.yml host_count=1 >>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin >>>>>> # - include: tasks/create-hosts.yml host_count=2 >>>>>> host_type=ambari_slave,search,ec2 >>>>>> - include: tasks/create-hosts.yml host_count=1 >>>>>> host_type=ambari_slave,search,ec2 >>>>>> - include: tasks/create-hosts.yml host_count=1 >>>>>> host_type=ambari_slave,web,ec2 >>>>>> <-- END SNIP --> >>>>>> >>>>>> I also had to change the line "ansible-playbook -vvv -i ec2.py >>>>>> playbook.yml \ >>>>>> --skip-tags="quick_dev,sensor-stubs" \ >>>>>> --extra-vars="env=$ENV" \ >>>>>> $EXTRA_ARGS" >>>>>> >>>>>> to "ansible-playbook -vvv -i ec2.py playbook.yml \ >>>>>> --skip-tags="quick_dev,sensor-stubs" \ >>>>>> --extra-vars="env=$ENV" \ >>>>>> $EXTRA_ARGS -c paramiko" >>>>>> in run.sh otherwise ssh would time out. >>>>>> >>>>>> However, Ansible errors out with: >>>>>> >>>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | TASK [ambari_config : >>>>>> Deploy cluster with Ambari; >>>>>> http://ec2-35-164-26-255.us-west-2.compute.amazonaws.com:8080] *** >>>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | task path: >>>>>> /home/laurens/SAPSource/metron/metron-deployment/roles/ >>>>>> ambari_config/tasks/main.yml:36 >>>>>> 2017-05-16 18:02:52,660 p=5362 u=laurens | fatal: >>>>>> [ec2-35-164-26-255.us-west-2.compute.amazonaws.com]: FAILED! => >>>>>> {"changed": false, "failed": true, "invocation": {"module_args": >>>>>> {"blueprint_name": "metron_blueprint", "blueprint_var": {"groups": >>>>>> [{"cardinality": 1, "components": [{"name": "NAMENODE"}, {"name": >>>>>> "SECONDARY_NAMENODE"}, {"name": "RESOURCEMANAGER"}, {"name": >>>>>> "HISTORYSERVER"}, {"name": "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, >>>>>> {"name": "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name": >>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name": "ES_SLAVE"}], >>>>>> "configuration": [], "name": "master_1"}, {"cardinality": 1, >>>>>> "components": [{"name": "ZOOKEEPER_SERVER"}, {"name": "NIMBUS"}, >>>>>> {"name": "STORM_UI_SERVER"}, {"name": "DRPC_SERVER"}, {"name": >>>>>> "SPARK_JOBHISTORYSERVER"}, {"name": "HBASE_MASTER"}, {"name": >>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name": >>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name": >>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name": >>>>>> "APP_TIMELINE_SERVER"}, {"name": "ES_SLAVE"}], "configuration": [], >>>>>> "name": "master_2"}, {"cardinality": 1, "components": [{"name": >>>>>> "METRON_INDEXING"}, {"name": "METRON_ENRICHMENT_MASTER"}, {"name": >>>>>> "METRON_PARSERS"}, {"name": "METRON_REST"}, {"name": "DATANODE"}, >>>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name": >>>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name": >>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name": >>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name": >>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [], >>>>>> "name": "metron"}, {"cardinality": 1, "components": [{"name": >>>>>> "KIBANA_MASTER"}, {"name": "ES_MASTER"}], "configuration": [], "name": >>>>>> "web"}, {"cardinality": "1+", "components": [{"name": "DATANODE"}, >>>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name": >>>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name": >>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name": >>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name": >>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [], >>>>>> "name": "slaves"}], "required_configurations": [{"metron-env": >>>>>> {"es_hosts": >>>>>> "ec2-35-164-136-88.us-west-2.compute.amazonaws.com,ec2-34-20 >>>>>> 8-94-45.us-west-2.compute.amazonaws.com", >>>>>> "metron_jdbc_driver": "org.h2.Driver", "metron_jdbc_password": "root", >>>>>> "metron_jdbc_platform": "h2", "metron_jdbc_url": >>>>>> "jdbc:h2:file:~/metrondb", "metron_jdbc_username": "root", >>>>>> "storm_rest_addr": >>>>>> "ec2-34-208-94-45.us-west-2.compute.amazonaws.com:8744", >>>>>> "zeppelin_server_url": >>>>>> "ec2-34-210-151-29.us-west-2.compute.amazonaws.com"}}, {"kibana-env": >>>>>> {"kibana_default_application": "dashboard/Metron-Dashboard", >>>>>> "kibana_es_url": >>>>>> "http://ec2-35-164-136-88.us-west-2.compute.amazonaws.com:9200";, >>>>>> "kibana_log_dir": "/var/log/kibana", "kibana_pid_dir": >>>>>> "/var/run/kibana", "kibana_server_port": 5000}}], "stack_name": "HDP", >>>>>> "stack_version": "2.5"}, "cluster_name": "metron", "cluster_state": >>>>>> "present", "configurations": [{"zoo.cfg": {"dataDir": >>>>>> "/data1/hadoop/zookeeper"}}, {"hadoop-env": {"dtnode_heapsize": 1024, >>>>>> "namenode_heapsize": 2048}}, {"hbase-env": {"hbase_master_heapsize": >>>>>> 1024, "hbase_regionserver_heapsize": 1024}}, {"hdfs-site": >>>>>> {"dfs.datanode.data.dir": >>>>>> "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data", >>>>>> "dfs.journalnode.edits.dir": "/data1/hadoop/hdfs/journalnode", >>>>>> "dfs.namenode.checkpoint.dir": "/data1/hadoop/hdfs/namesecondary", >>>>>> "dfs.namenode.name.dir": "/data1/hadoop/hdfs/namenode"}}, >>>>>> {"mapred-site": {"mapreduce.jobhistory.recovery.store.leveldb.path": >>>>>> "/data1/hadoop/mapreduce/jhs", "mapreduce.map.memory.mb": 1229, >>>>>> "mapreduce.reduce.memory.mb": 1229}}, {"yarn-site": >>>>>> {"yarn.nodemanager.local-dirs": "/data1/hadoop/yarn/local", >>>>>> "yarn.nodemanager.log-dirs": "/data1/hadoop/yarn/log", >>>>>> "yarn.nodemanager.resource.memory-mb": 4096, >>>>>> "yarn.timeline-service.leveldb-state-store.path": >>>>>> "/data1/hadoop/yarn/timeline", >>>>>> "yarn.timeline-service.leveldb-timeline-store.path": >>>>>> "/data1/hadoop/yarn/timeline"}}, {"storm-site": {"storm.local.dir": >>>>>> "/data1/hadoop/storm", "supervisor.slots.ports": "[6700, 6701, 6702, >>>>>> 6703]", "topology.classpath": "/etc/hbase/conf:/etc/hadoop/conf"}}, >>>>>> {"kafka-broker": {"log.dirs": "/data2/kafka-log"}}, {"metron-env": >>>>>> {"parsers": "bro,snort,yaf"}}, {"elastic-site": >>>>>> {"gateway_recover_after_data_nodes": 1, "index_number_of_replicas": >>>>>> 1, >>>>>> "index_number_of_shards": 2, "network_host": "[ _local_, _eth0_ ]", >>>>>> "zen_discovery_ping_unicast_hosts": "[ >>>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com, >>>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com ]"}}], "host": >>>>>> "ec2-35-164-26-255.us-west-2.compute.amazonaws.com", "password": >>>>>> "admin", "port": 8080, "username": "admin", "wait_for_complete": >>>>>> true}, "module_name": "ambari_cluster_state"}, "msg": "Ambari client >>>>>> exception occurred: Could not create cluster: request code 400, >>>>>> request message {\n \"status\" : 400,\n \"message\" : >>>>>> \"Invalid Cluster Creation Template: >>>>>> org.apache.ambari.server.topology.InvalidTopologyTemplateException: >>>>>> Host group 'slaves' must contain at least one 'hosts/fqdn' or a >>>>>> 'host_count' value\"\n}"} >>>>>> >>>>>> PLAY RECAP ****************************** >>>>>> ****************************** >>>>>> ********* >>>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com : ok=49 changed=27 >>>>>> unreachable=0 failed=0 >>>>>> ec2-34-209-224-190.us-west-2.compute.amazonaws.com : ok=23 >>>>>> changed=8 >>>>>> unreachable=0 failed=0 >>>>>> ec2-34-210-151-29.us-west-2.compute.amazonaws.com : ok=58 >>>>>> changed=31 >>>>>> unreachable=0 failed=0 >>>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com : ok=49 >>>>>> changed=27 >>>>>> unreachable=0 failed=0 >>>>>> ec2-35-164-26-255.us-west-2.compute.amazonaws.com : ok=54 >>>>>> changed=28 >>>>>> unreachable=0 failed=1 >>>>>> localhost : ok=29 changed=12 unreachable=0 >>>>>> failed=0 >>>>>> >>>>>> Any idea what might be going on? >>>>>> >>>>>> (Don't worry about the URLs, the instances have been terminated) >>>>>> >>>>>> >>>>> >
