Hi David,

That works, thanks! Now trying to get this down even more (4 sounds like a nice number :p)

On 2017-05-30 17:05, David Lyle wrote:
This one should do the trick for you.

Due to how smal_cluster.yml is structured (designed for 10+ nodes)- you'll
only get 2 data nodes, so you'll have to reduce replication to 2 if you
don't want to see under-replicated block warnings.

    - include: tasks/create-hosts.yml host_count=1
host_type=sensors,ambari_master,ec2,monit
    - include: tasks/create-hosts.yml host_count=2
host_type=ambari_slave,search,ec2
    - include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,ec2
    - include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,enrichment,metron,ec2,zeppelin
    - include: tasks/create-hosts.yml host_count=1
host_type=pcap_server,monit,ambari_slave,web,ec2

The reason the first one didn't work is because order matters. In the that
example, the host created by

    - include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,enrichment,metron,ec2,zeppelin

was ending up in 2 separate hostgroups, which is not allowed. Relocating that line solves the problem. I also relocated the web hosts (ES Master and
Kibana) since there were fewer hosts.


-D...


On Tue, May 30, 2017 at 12:49 PM, Laurens Vets <laur...@daemon.be> wrote:

I put in what you said previously :)

- include: tasks/create-hosts.yml host_count=1
host_type=sensors,ambari_master,ec2,monit
- include: tasks/create-hosts.yml host_count=1
host_type=pcap_server,monit,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,enrichment,metron,ec2,zeppelin
- include: tasks/create-hosts.yml host_count=2
host_type=ambari_slave,search,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,web,ec2

On 2017-05-30 08:53, David Lyle wrote:

That's actually a different error than you posted before. That error is Ambari telling you that the blueprint that Ansible submitted is invalid. Likely a result of a required hostgroup not being defined. Can you (re?)
post what you've put in amazon-ec2/playbook.yml?

-D...


On Mon, May 29, 2017 at 12:02 PM, Laurens Vets <laur...@daemon.be> wrote:

Same problem as all the previous ones...

TASK [ambari_config : Install python-requests]
*********************************
ok: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com] => {"attempts":
1, "changed": false, "msg": "", "rc": 0, "results":
["python-requests-2.6.0-3.el6.noarch providing python-requests is
already
installed"]}

TASK [ambari_config : check if ambari-server is up on
ec2-34-209-23-182.us-west-2.compute.amazonaws.com:8080] ***
ok: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com] => {"changed": false, "elapsed": 120, "path": null, "port": 8080, "search_regex": null,
"state": "started"}

TASK [ambari_config : Deploy cluster with Ambari;
http://ec2-34-209-23-182.us-west-2.compute.amazonaws.com:8080] ***
fatal: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com]: FAILED! =>
{"changed": false, "failed": true, "msg": "Ambari client exception
occurred: Could not create cluster: request code 400,
 request message {\n  \"status\" : 400,\n  \"message\" : \"Topology
validation failed: org.apache.ambari.server.topol
ogy.InvalidTopologyException:
The following hosts are mapped to multiple host groups: [
ec2-34-208-74-17.us-west-2.compute.amazonaws.com]. Be aware that host
names are converted to lowercase, case differences do not matter in
Ambari
deployments.\"\n}"}
        to retry, use: --limit @/root/metron/metron-deploymen
t/amazon-ec2/playbook.retry

PLAY RECAP ************************************************************
*********
ec2-34-208-74-17.us-west-2.compute.amazonaws.com : ok=47 changed=31
 unreachable=0    failed=0
ec2-34-209-23-182.us-west-2.compute.amazonaws.com : ok=48 changed=28
 unreachable=0    failed=1
ec2-35-163-138-4.us-west-2.compute.amazonaws.com : ok=41 changed=27
 unreachable=0    failed=0
ec2-35-163-150-6.us-west-2.compute.amazonaws.com : ok=41 changed=27
 unreachable=0    failed=0
ec2-35-167-207-80.us-west-2.compute.amazonaws.com : ok=17 changed=8
unreachable=0    failed=0
ec2-54-70-75-61.us-west-2.compute.amazonaws.com : ok=41   changed=27
 unreachable=0    failed=0
localhost                  : ok=18   changed=13   unreachable=0
failed=0

root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#


On 2017-05-27 04:45, David Lyle wrote:

For some reason, kibana_hosts is undefined, so the curl command that
loads
the Kibana dashboard doesn't contain the URL for Kibana. If you look at small_cluster.yml, you'll see that the Kibana should have been installed
on
the first host in the "web" hostgroup. Verify that you placed a host in that hostgroup (it's the 5th line in Mike's example). If you did, check that host on Ambari and verify that Kibana was installed, if not, you'll
want to dig through the install log to find out why.

Also- now that I've looked at the example carefully, I noticed that the Zeppelin hostgroup was duplicated. That's not a problem, because the 2nd host will be ignored, maybe you caught it when you created your own.

I would try something like (for a 6 node host cluster):

    - include: tasks/create-hosts.yml host_count=1
host_type=sensors,ambari_
master,ec2,monit
    - include: tasks/create-hosts.yml host_count=1
host_type=pcap_server,monit,ec2
    - include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,
enrichment,metron,ec2,zeppelin
    - include: tasks/create-hosts.yml host_count=2
host_type=ambari_slave,search,ec2
    - include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,web,ec2

-D...


On Fri, May 26, 2017 at 8:15 PM, Laurens Vets <laur...@daemon.be>
wrote:

Different error than before, but it still fails:


TASK [ambari_gather_facts : set_fact] ******************************
************
ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
{"ansible_facts": {"kibana_hosts": []}, "changed": false}

TASK [ambari_gather_facts : debug] ******************************
***************
ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] => {
    "msg": "zookeeper_port = 2181, zookeeper_hosts = [u'
ec2-52-43-143-124.us-west-2.compute.amazonaws.com'], zookeeper_url =
ec2-52-43-143-124.us-west-2.compute.amazonaws.com:2181,
kafka_broker_port
= 6667, kafka_broker_hosts = [u'ec2-34-210-18-244.us-west-2
.compute.amazonaws.com', u'ec2-35-164-236-192.us-west-2
.compute.amazonaws.com', u'ec2-52-41-30-101.us-west-2.c
ompute.amazonaws.com'], kafka_broker_url =
ec2-34-210-18-244.us-west-2.co
mpute.amazonaws.com:6667,ec2-35-164-236-192.us-west-2.compu
te.amazonaws.com:6667,ec2-52-41-30-101.us-west-2.compute.
amazonaws.com:6667, metron_hosts = [u'ec2-35-164-236-192.us-west-
2.compute.amazonaws.com']"
}

TASK [load_web_templates : Load ES Templates]
**********************************
changed: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
{"changed": true, "cmd": ["curl", "-s", "-w", "%{http_code}", "-u", "admin:admin", "-H", "X-Requested-By: ambari", "-X", "POST", "-d", "{
\"RequestInfo\": { \"context\": \"Install ES Template from REST\",
\"command\": \"ELASTICSEARCH_TEMPLATE_INSTALL\"},\"Requests/resource_
filters\":
[{\"service_name\": \"METRON\",\"component_name\":
\"METRON_INDEXING\",\"hosts\" : \"ec2-35-164-236-192.us-west-2
.compute.amazonaws.com\"}]}", "http://ec2-34-210-207-190.us-
west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests"],
"delta": "0:00:00.049367", "end": "2017-05-26 21:33:34.767106",
"failed":
false, "failed_when_result": false, "rc": 0, "start": "2017-05-26
21:33:34.717739", "stderr": "", "stdout": "{\n  \"href\" : \"
http://ec2-34-210-207-190.us-west-2.compute.amazonaws.com:
8080/api/v1/clusters/metron/requests/14\",\n  \"Requests\" : {\n
\"id\" : 14,\n    \"status\" : \"Accepted\"\n  }\n}202",
"stdout_lines":
["{", "  \"href\" : \"http://ec2-34-210-207-190.us
-west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14\
<http://west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14%5C>
<http://west-2.compute.amazonaws.com:8080/api/v1/clusters/
metron/requests/14%5C>

",",
"  \"Requests\" : {", "    \"id\" : 14,", "    \"status\" :
\"Accepted\"",
"  }", "}202"], "warnings": []}

TASK [load_web_templates : Load Kibana Dashboard]
******************************
fatal: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com]: FAILED!
=>
{"failed": true, "msg": "the field 'args' has an invalid value, which appears to include a variable that is undefined. The error was: list
object
has no element 0\n\nThe error appears to have been in
'/root/metron/metron-deployment/roles/load_web_templates/
tasks/main.yml':
line 26, column 3, but may\nbe elsewhere in the file depending on the
exact
syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Load
Kibana Dashboard\n  ^ here\n"}
        to retry, use: --limit @/root/metron/metron-deploymen
t/amazon-ec2/playbook.retry

PLAY RECAP ******************************
******************************
*********
ec2-34-210-18-244.us-west-2.compute.amazonaws.com : ok=41 changed=27
 unreachable=0    failed=0
ec2-34-210-207-190.us-west-2.compute.amazonaws.com : ok=80
 changed=31
 unreachable=0    failed=1
ec2-34-211-19-168.us-west-2.compute.amazonaws.com : ok=41 changed=27
 unreachable=0    failed=0
ec2-35-164-236-192.us-west-2.compute.amazonaws.com : ok=47
 changed=31
 unreachable=0    failed=0
ec2-52-41-30-101.us-west-2.compute.amazonaws.com : ok=41 changed=27
 unreachable=0    failed=0
ec2-52-43-143-124.us-west-2.compute.amazonaws.com : ok=41 changed=27
 unreachable=0    failed=0
localhost                  : ok=16   changed=11   unreachable=0
failed=0

root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#


On 2017-05-26 11:59, Michael Miklavcic wrote:

Maybe you could re-purpose the playbook.yml for the 10-node install -

comment out some of the host creation lines and move the host types
where
you want them installed, e.g.

From this

  tasks:
    - include: tasks/create-keypair.yml
    - include: tasks/create-vpc.yml
    - include: tasks/create-open-inbound-security-group.yml
    - include: tasks/create-open-outbound-security-group.yml
    - include: tasks/create-hosts.yml host_count=1
host_type=sensors,ambari_master,ec2,monit
    - include: tasks/create-hosts.yml host_count=4
host_type=ambari_slave,ec2
    - include: tasks/create-hosts.yml host_count=1
host_type=pcap_server,monit,ec2
    - include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,enrichment,metron,ec2,zeppelin
    - include: tasks/create-hosts.yml host_count=2
host_type=ambari_slave,search,ec2
    - include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,web,ec2

to this (6-node)

  tasks:
    - include: tasks/create-keypair.yml
    - include: tasks/create-vpc.yml
    - include: tasks/create-open-inbound-security-group.yml
    - include: tasks/create-open-outbound-security-group.yml
    - include: tasks/create-hosts.yml host_count=1
host_type=sensors,ambari_master,ec2,monit,zeppelin,search,web
    - include: tasks/create-hosts.yml host_count=3
host_type=ambari_slave,ec2
    - include: tasks/create-hosts.yml host_count=1 host_type=
ambari_slave
,pcap_server,monit,ec2
    - include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,enrichment,metron,ec2,zeppelin

Of course, this is just to demonstrate re-locating items, not
necessarily
how you would want to lay them out.

Best,
Mike Miklavcic


On Fri, May 26, 2017 at 11:28 AM, Laurens Vets <laur...@daemon.be>
wrote:

I'm trying to install Metron in AWS but with fewer hosts than the
default

10 node cluster install... So far no luck. Whatever I do, it always
ends
in
errors...


On 2017-05-26 10:16, David Lyle wrote:

I'm gonna to venture a 'no' simply because I don't know what that is.

Afaik, single_node_vm.yml was never used with AWS. What are you
trying
to
do?

-D...


On Thu, May 25, 2017 at 8:50 PM, Laurens Vets <laur...@daemon.be>
wrote:

Is AWS single_node_vm still supported?

Reply via email to