I'm not quite following. Unfortunately, I removed the previous ec2
instances and started again to test. Same error for host
ec2-52-40-75-137.us-west-2.compute.amazonaws.com with the following
configuration:
- include: tasks/create-hosts.yml host_count=1
host_type=sensors,ambari_master,ec2,monit
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=pcap_server,monit,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,enrichment,metron,ec2,zeppelin
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,search,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,web,ec2
The AWS mapping:
[metron-test] ambari_slave,search,ec2
ec2-35-162-147-217.us-west-2.compute.amazonaws.com
[metron-test] sensors,ambari_master,ec2,monit
ec2-52-40-75-137.us-west-2.compute.amazonaws.com [FAILED]
[metron-test] ambari_slave,web,ec2
ec2-52-10-238-174.us-west-2.compute.amazonaws.com
[metron-test] ambari_slave,enrichment,metron,ec2,zeppelin
ec2-54-148-199-7.us-west-2.compute.amazonaws.com
[metron-test] ambari_slave,ec2
ec2-54-71-23-118.us-west-2.compute.amazonaws.com
[metron-test] pcap_server,monit,ec2
ec2-35-161-165-22.us-west-2.compute.amazonaws.com
Log entries:
ASK [ambari_config : Deploy cluster with Ambari;
http://ec2-52-40-75-137.us-west-2.compute.amazonaws.com:8080] ***
fatal: [ec2-52-40-75-137.us-west-2.compute.amazonaws.com]: FAILED! =>
{"changed": false, "failed": true, "msg": "Ambari client exception
occurred: Could not create cluster: request code 400,
request message {\n \"status\" : 400,\n \"message\" : \"Topology
validation failed:
org.apache.ambari.server.topology.InvalidTopologyException: The
following hosts are mapped to multiple host groups:
[ec2-54-148-199-7.us-west-2.compute.amazonaws.com]. Be aware that host
names are converted to lowercase, case differences do not matter in
Ambari deployments.\"\n}"}
to retry, use: --limit
@/root/metron/metron-deployment/amazon-ec2/playbook.retry
PLAY RECAP
*********************************************************************
ec2-35-161-165-22.us-west-2.compute.amazonaws.com : ok=17 changed=8
unreachable=0 failed=0
ec2-35-162-147-217.us-west-2.compute.amazonaws.com : ok=41 changed=27
unreachable=0 failed=0
ec2-52-10-238-174.us-west-2.compute.amazonaws.com : ok=41 changed=27
unreachable=0 failed=0
ec2-52-40-75-137.us-west-2.compute.amazonaws.com : ok=48 changed=28
unreachable=0 failed=1
ec2-54-148-199-7.us-west-2.compute.amazonaws.com : ok=47 changed=31
unreachable=0 failed=0
ec2-54-71-23-118.us-west-2.compute.amazonaws.com : ok=41 changed=27
unreachable=0 failed=0
localhost : ok=20 changed=15 unreachable=0
failed=0
The instances have been disabled again for now. Once I get it up and
running, I'll probably move the production stuff into Terraform with
Ansible. This is purely for testing purposes.
On 2017-05-25 10:34, Nick Allen wrote:
> What host groups did ec2-34-210-245-155.us-west-2.compute.amazonaws.com [8]
> get assigned to? You can look at the labels that get attached to the host in
> the EC2 web interface.
>
> I have a feeling it has something to do with the Ambari blueprint that is
> used (see metron-deployment/roles/ambari_config/vars/small_cluster.yml). You
> probably need to customize or define your own blueprint.
>
> WARNING: Please know that the automated EC2 setup is only for disposable,
> short-lived, development/testing purposes. It is not at all secure.
>
> If you are wanting to run Metron in AWS for any period of time, a better
> approach is to define your VPC, spin up your EC2 hosts, install Ambari, then
> use Metron's MPack to install Metron.
>
> On Thu, May 25, 2017 at 1:00 PM, Laurens Vets <[email protected]> wrote:
>
>> Deploying the standard 10 instance setup works. However, for our current
>> needs, 10 m4.xlarge instances seem overkill and we want to deploy Metron on
>> only 5 hosts for now.
>>
>> I would think that editing metron/metron-deployment/amazon-ec2/playbook.yml
>> would be enough. I changed the following:
>>
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=sensors,ambari_master,ec2,monit
>> - include: tasks/create-hosts.yml host_count=4 host_type=ambari_slave,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=pcap_server,monit,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>> - include: tasks/create-hosts.yml host_count=2
>> host_type=ambari_slave,search,ec2
>> - include: tasks/create-hosts.yml host_count=1 host_type=ambari_slave,web,ec2
>>
>> to:
>>
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=sensors,ambari_master,ec2,monit
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=pcap_server,monit,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>> - include: tasks/create-hosts.yml host_count=2 host_type=ambari_slave,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=ambari_slave,web,search,ec2
>>
>> But now deployment fails with:
>>
>> TASK [ambari_config : Install python-requests]
>> *********************************
>> ok: [ec2-34-211-7-200.us-west-2.compute.amazonaws.com [1]] => {"attempts":
>> 1, "changed": false, "msg": "", "rc": 0, "results":
>> ["python-requests-2.6.0-3.el6.noarch providing python-requests is already
>> installed"]}
>>
>> TASK [ambari_config : check if ambari-server is up on
>> ec2-34-211-7-200.us-west-2.compute.amazonaws.com:8080 [2]] ***
>> ok: [ec2-34-211-7-200.us-west-2.compute.amazonaws.com [1]] => {"changed":
>> false, "elapsed": 120, "path": null, "port": 8080, "search_regex": null,
>> "state": "started"}
>>
>> TASK [ambari_config : Deploy cluster with Ambari;
>> http://ec2-34-211-7-200.us-west-2.compute.amazonaws.com:8080 [2]] ***
>> fatal: [ec2-34-211-7-200.us-west-2.compute.amazonaws.com [1]]: FAILED! =>
>> {"changed": false, "failed": true, "msg": "Ambari client exception occurred:
>> Could not create cluster: request code 400, request
>> message {\n \"status\" : 400,\n \"message\" : \"Topology validation
>> failed: org.apache.ambari.server.topology.InvalidTopologyException: The
>> following hosts are mapped to multiple host groups:
>> [ec2-34-210-245-155.us-west-2.compute.amazonaws.com [3]]. Be aware that host
>> names are converted to lowercase, case differences do not matter in Ambari
>> deployments.\"\n}"}
>> to retry, use: --limit
>> @/root/metron/metron-deployment/amazon-ec2/playbook.retry
>>
>> PLAY RECAP
>> *********************************************************************
>> ec2-34-210-137-42.us-west-2.compute.amazonaws.com [4] : ok=41 changed=27
>> unreachable=0 failed=0
>> ec2-34-210-245-155.us-west-2.compute.amazonaws.com [3] : ok=47 changed=31
>> unreachable=0 failed=0
>> ec2-34-211-3-80.us-west-2.compute.amazonaws.com [5] : ok=17 changed=8
>> unreachable=0 failed=0
>> ec2-34-211-7-200.us-west-2.compute.amazonaws.com [1] : ok=48 changed=28
>> unreachable=0 failed=1
>> ec2-35-165-165-255.us-west-2.compute.amazonaws.com [6] : ok=41 changed=27
>> unreachable=0 failed=0
>> ec2-54-70-66-181.us-west-2.compute.amazonaws.com [7] : ok=41 changed=27
>> unreachable=0 failed=0
>> localhost : ok=18 changed=13 unreachable=0 failed=0
>>
>> Any idea what might be going on? Did I miss a setting somewhere else?
Links:
------
[1] http://ec2-34-211-7-200.us-west-2.compute.amazonaws.com
[2] http://ec2-34-211-7-200.us-west-2.compute.amazonaws.com:8080
[3] http://ec2-34-210-245-155.us-west-2.compute.amazonaws.com
[4] http://ec2-34-210-137-42.us-west-2.compute.amazonaws.com
[5] http://ec2-34-211-3-80.us-west-2.compute.amazonaws.com
[6] http://ec2-35-165-165-255.us-west-2.compute.amazonaws.com
[7] http://ec2-54-70-66-181.us-west-2.compute.amazonaws.com
[8] http://ec2-34-210-245-155.us-west-2.compute.amazonaws.com/
