[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/436


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104698233
  
--- Diff: metron-deployment/roles/kibana/README.md ---
@@ -1,35 +0,0 @@
-Kibana 4
-
-
-This role installs Kibana along with the default Metron Dashboard.
-
-### FAQ
-
- How do I change Metron's default dashboard?
--- End diff --

Yeah, I'm good either way, I thought it may help you. If it doesn't, no 
point to rushing it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104678359
  
--- Diff: metron-deployment/roles/kibana/README.md ---
@@ -1,35 +0,0 @@
-Kibana 4
-
-
-This role installs Kibana along with the default Metron Dashboard.
-
-### FAQ
-
- How do I change Metron's default dashboard?
--- End diff --

I'm inclined (possibly by my own self interest) to make it a follow on jira 
that gets resolved either before or after 745.  745 takes care of this file 
(which is actually why I thought of it).

I definitely agree on using the same file though, but I'm not sure off the 
top of my head how much refactoring happens there.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104676140
  
--- Diff: metron-deployment/roles/kibana/README.md ---
@@ -1,35 +0,0 @@
-Kibana 4
-
-
-This role installs Kibana along with the default Metron Dashboard.
-
-### FAQ
-
- How do I change Metron's default dashboard?
--- End diff --

That's a really good catch. I'd recommend opening a separate Jira to use 
the same pickle file that the Ambari deployment uses. Preferably before 
METRON-745. With a duplicate file, we run the risk of someone like me 
accidentally desyncing the dashboards. Thoughts? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104673955
  
--- Diff: metron-deployment/roles/kibana/README.md ---
@@ -1,35 +0,0 @@
-Kibana 4
-
-
-This role installs Kibana along with the default Metron Dashboard.
-
-### FAQ
-
- How do I change Metron's default dashboard?
--- End diff --

kibana-index.json still exists in the docker stuff. Given that the module 
is buried in the Ambari stuff, do we still need/want the instructions and an 
elasticdump done to update that?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104673176
  
--- Diff: metron-deployment/roles/ambari_config/vars/single_node_vm.yml ---
@@ -80,10 +88,32 @@ configurations:
   - kafka-broker:
   log.dirs: '{{ kafka_log_dirs }}'
   delete.topic.enable: "true"
+  - metron-env:
+  parsers: "bro,snort"
+  - elastic-site:
+  index_number_of_shards: 1
+  index_number_of_replicas: 0
+  zen_discovery_ping_unicast_hosts: "{{ groups.search | join(',') }}"
+  gateway_recover_after_data_nodes: 1
+  network_host: "_lo_,_eth0_,_eth1_"
+  masters_also_are_datanodes: "1"
--- End diff --

Heh, you know, that's EXACTLY where I was with it. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104672196
  
--- Diff: metron-deployment/extra_modules/ambari_service_state.py ---
@@ -0,0 +1,352 @@
+#!/usr/bin/python
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+DOCUMENTATION = '''
+---
+module: ambari_service_state
+version_added: "2.1"
+author: Apache Metron (Incubating : 
https://github.com/apache/incubator-metron )
+short_description: Start/Stop/Change Service or Component State
+description:
+- Start/Stop/Change Service or Component State
+options:
+  host:
+description:
+  The hostname for the ambari web server
+  port:
+description:
+  The port for the ambari web server
+  username:
+description:
+  The username for the ambari web server
+  password:
+description:
+  The name of the cluster in web server
+required: yes
+  cluster_name:
+description:
+  The name of the cluster in ambari
+required: yes
+  service_name:
+description:
+  The name of the service to alter
+required: no
+  component_name:
+description:
+  The name of the component to alter
+required: no
+  component_host:
+description:
+  The host running the targeted component. Required when 
component_name is used.
+required: no
+  state:
+description:
+  The desired service/component state.
+  wait_for_complete:
+description:
+  Whether to wait for the request to complete before returning. 
Default is False.
+required: no
+  requirements: [ 'requests']
+'''
+
+EXAMPLES = '''
+# must use full relative path to any files in stored in 
roles/role_name/files/
+- name: Create a new ambari cluster
+ambari_cluster_state:
+  host: localhost
+  port: 8080
+  username: admin
+  password: admin
+  cluster_name: my_cluster
+  cluster_state: present
+  blueprint_var: roles/my_role/files/blueprint.yml
+  blueprint_name: hadoop
+  wait_for_complete: True
+- name: Start the ambari cluster
+  ambari_cluster_state:
+host: localhost
+port: 8080
+username: admin
+password: admin
+cluster_name: my_cluster
+cluster_state: started
+wait_for_complete: True
+- name: Stop the ambari cluster
+  ambari_cluster_state:
+host: localhost
+port: 8080
+username: admin
+password: admin
+cluster_name: my_cluster
+cluster_state: stopped
+wait_for_complete: True
+- name: Delete the ambari cluster
+  ambari_cluster_state:
+host: localhost
+port: 8080
+username: admin
+password: admin
+cluster_name: my_cluster
+cluster_state: absent
+'''
+
+RETURN = '''
+results:
+description: The content of the requests object returned from the 
RESTful call
+returned: success
+type: string
+'''
+
+__author__ = 'apachemetron'
+
+import json
+
+try:
+import requests
+except ImportError:
+REQUESTS_FOUND = False
+else:
+REQUESTS_FOUND = True
+
+
+def main():
+
+argument_spec = dict(
+host=dict(type='str', default=None, required=True),
+port=dict(type='int', default=None, required=True),
+username=dict(type='str', default=None, required=True),
+password=dict(type='str', default=None, required=True),
+cluster_name=dict(type='str', default=None, required=True),
+state=dict(type='str', default=None, required=True,
+   choices=['started', 'stopped', 'deleted']),
+service_name=dict(type='str', required=False),
+component_name=dict(type='str', default=None, required=False),
+component_host=dict(type='str', 

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104671925
  
--- Diff: metron-deployment/roles/ambari_config/vars/single_node_vm.yml ---
@@ -80,10 +88,32 @@ configurations:
   - kafka-broker:
   log.dirs: '{{ kafka_log_dirs }}'
   delete.topic.enable: "true"
+  - metron-env:
+  parsers: "bro,snort"
+  - elastic-site:
+  index_number_of_shards: 1
+  index_number_of_replicas: 0
+  zen_discovery_ping_unicast_hosts: "{{ groups.search | join(',') }}"
+  gateway_recover_after_data_nodes: 1
+  network_host: "_lo_,_eth0_,_eth1_"
+  masters_also_are_datanodes: "1"
--- End diff --

I'm fine with whatever works.  It's ES configuration, so if it wants to 
accept "1", it can be my guest.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104667214
  
--- Diff: metron-deployment/roles/quick_dev/tasks/main.yml ---
@@ -15,23 +15,50 @@
 #  limitations under the License.
 #
 ---
-#
-# Workaround for Kafka not starting
-# Fire off async start followed by
-# Sync start -execution will pause until
-# final start completes.
-#
-- name: Start the ambari cluster - no wait
-  ambari_cluster_state:
+- name: Delete the Metron Components from Ambari
+  ambari_service_state:
 host: "{{ groups.ambari_master[0] }}"
 port: "{{ ambari_port }}"
 username: "{{ ambari_user }}"
 password: "{{ ambari_password }}"
 cluster_name: "{{ cluster_name }}"
-cluster_state: started
-wait_for_complete: False
+state: deleted
+component_name: "{{ item }}"
+component_host: "{{ inventory_hostname }}"
+  with_items:
+- METRON_ENRICHMENT_MASTER
+- METRON_INDEXING
+- METRON_PARSERS
+
+- name: Remove the Metron packages
+  package:
+name: "{{ item }}"
+state: absent
+  with_items:
+- metron-common
+- metron-data-management
+- metron-parsers
+- metron-enrichment
+- metron-indexing
+- metron-elasticsearch
+
+- name: Re-install the Metron Packages via Ambari
--- End diff --

No, I want them to exist there and they do. It simulates the old 
run_enrichment_role task by re-installing the Metron rpms and reloads configs, 
but leaves everything else alone.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104666299
  
--- Diff: metron-deployment/roles/ambari_config/vars/single_node_vm.yml ---
@@ -80,10 +88,32 @@ configurations:
   - kafka-broker:
   log.dirs: '{{ kafka_log_dirs }}'
   delete.topic.enable: "true"
+  - metron-env:
+  parsers: "bro,snort"
+  - elastic-site:
+  index_number_of_shards: 1
+  index_number_of_replicas: 0
+  zen_discovery_ping_unicast_hosts: "{{ groups.search | join(',') }}"
+  gateway_recover_after_data_nodes: 1
+  network_host: "_lo_,_eth0_,_eth1_"
+  masters_also_are_datanodes: "1"
--- End diff --

Scratch the scratch. It needs to be "1" or ""true"" for now. It's the same 
ES problem we talked about above. If you want me to look at move the "translate 
True to ""true"" " test to params, I can do that, but it's going to need to 
exist somewhere. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104666805
  
--- Diff: metron-deployment/roles/ambari_master/tasks/main.yml ---
@@ -38,6 +38,16 @@
   register: ambari_server_setup
   failed_when: ambari_server_setup.stderr
 
+- name: Copy MPack to Ambari Host
+  copy:
+src: "{{ playbook_dir 
}}/../packaging/ambari/metron-mpack/target/metron_mpack-0.3.1.0.tar.gz"
--- End diff --

Sure, good suggestion. In test now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104665126
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml
 ---
@@ -27,6 +27,14 @@
 Cluster name identifies your cluster
 
 
+masters_also_are_datanodes
+"false"
--- End diff --

No, it's not that important.  Can you add to the description that it has to 
be in quotes for ES compatibility?  I can definitely see that causing confusion 
otherwise.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104663778
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml
 ---
@@ -27,6 +27,14 @@
 Cluster name identifies your cluster
 
 
+masters_also_are_datanodes
+"false"
--- End diff --

Just looked at it again, it was a quick hack to overcome a silly bug in ES 
where False != false. Ambari does some tricks under the hood with bools where 
they come out as True/False causing ES to fail. I can struggle with it some 
more if it's important to you, but I think it's going to end with 

> if False then value = "false" else value = "true"

 somewhere in the code. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104662631
  
--- Diff: metron-deployment/roles/ambari_config/vars/single_node_vm.yml ---
@@ -80,10 +88,32 @@ configurations:
   - kafka-broker:
   log.dirs: '{{ kafka_log_dirs }}'
   delete.topic.enable: "true"
+  - metron-env:
+  parsers: "bro,snort"
+  - elastic-site:
+  index_number_of_shards: 1
+  index_number_of_replicas: 0
+  zen_discovery_ping_unicast_hosts: "{{ groups.search | join(',') }}"
+  gateway_recover_after_data_nodes: 1
+  network_host: "_lo_,_eth0_,_eth1_"
+  masters_also_are_datanodes: "1"
--- End diff --

Scratch that, that's not an ES setting (they have some weirdness with 
proper booleans). I can try it and get back to you. Fwiw, 1 does work.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104662413
  
--- Diff: metron-deployment/roles/ambari_config/vars/single_node_vm.yml ---
@@ -80,10 +88,32 @@ configurations:
   - kafka-broker:
   log.dirs: '{{ kafka_log_dirs }}'
   delete.topic.enable: "true"
+  - metron-env:
+  parsers: "bro,snort"
+  - elastic-site:
+  index_number_of_shards: 1
+  index_number_of_replicas: 0
+  zen_discovery_ping_unicast_hosts: "{{ groups.search | join(',') }}"
+  gateway_recover_after_data_nodes: 1
+  network_host: "_lo_,_eth0_,_eth1_"
+  masters_also_are_datanodes: "1"
--- End diff --

1 works, ""true"" will work. It will fail on a proper boolean, so I left it 
as 1.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104662261
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml
 ---
@@ -27,6 +27,14 @@
 Cluster name identifies your cluster
 
 
+masters_also_are_datanodes
+"false"
--- End diff --

No. ES will fail on a bool.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104659900
  
--- Diff: metron-deployment/roles/ambari_master/tasks/main.yml ---
@@ -38,6 +38,16 @@
   register: ambari_server_setup
   failed_when: ambari_server_setup.stderr
 
+- name: Copy MPack to Ambari Host
+  copy:
+src: "{{ playbook_dir 
}}/../packaging/ambari/metron-mpack/target/metron_mpack-0.3.1.0.tar.gz"
--- End diff --

Can we pull the mpack version out into a variable so there are less places 
to change it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104654194
  
--- Diff: metron-deployment/extra_modules/ambari_service_state.py ---
@@ -0,0 +1,352 @@
+#!/usr/bin/python
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+DOCUMENTATION = '''
+---
+module: ambari_service_state
+version_added: "2.1"
+author: Apache Metron (Incubating : 
https://github.com/apache/incubator-metron )
+short_description: Start/Stop/Change Service or Component State
+description:
+- Start/Stop/Change Service or Component State
+options:
+  host:
+description:
+  The hostname for the ambari web server
+  port:
+description:
+  The port for the ambari web server
+  username:
+description:
+  The username for the ambari web server
+  password:
+description:
+  The name of the cluster in web server
+required: yes
+  cluster_name:
+description:
+  The name of the cluster in ambari
+required: yes
+  service_name:
+description:
+  The name of the service to alter
+required: no
+  component_name:
+description:
+  The name of the component to alter
+required: no
+  component_host:
+description:
+  The host running the targeted component. Required when 
component_name is used.
+required: no
+  state:
+description:
+  The desired service/component state.
+  wait_for_complete:
+description:
+  Whether to wait for the request to complete before returning. 
Default is False.
+required: no
+  requirements: [ 'requests']
+'''
+
+EXAMPLES = '''
+# must use full relative path to any files in stored in 
roles/role_name/files/
+- name: Create a new ambari cluster
+ambari_cluster_state:
+  host: localhost
+  port: 8080
+  username: admin
+  password: admin
+  cluster_name: my_cluster
+  cluster_state: present
+  blueprint_var: roles/my_role/files/blueprint.yml
+  blueprint_name: hadoop
+  wait_for_complete: True
+- name: Start the ambari cluster
+  ambari_cluster_state:
+host: localhost
+port: 8080
+username: admin
+password: admin
+cluster_name: my_cluster
+cluster_state: started
+wait_for_complete: True
+- name: Stop the ambari cluster
+  ambari_cluster_state:
+host: localhost
+port: 8080
+username: admin
+password: admin
+cluster_name: my_cluster
+cluster_state: stopped
+wait_for_complete: True
+- name: Delete the ambari cluster
+  ambari_cluster_state:
+host: localhost
+port: 8080
+username: admin
+password: admin
+cluster_name: my_cluster
+cluster_state: absent
+'''
+
+RETURN = '''
+results:
+description: The content of the requests object returned from the 
RESTful call
+returned: success
+type: string
+'''
+
+__author__ = 'apachemetron'
+
+import json
+
+try:
+import requests
+except ImportError:
+REQUESTS_FOUND = False
+else:
+REQUESTS_FOUND = True
+
+
+def main():
+
+argument_spec = dict(
+host=dict(type='str', default=None, required=True),
+port=dict(type='int', default=None, required=True),
+username=dict(type='str', default=None, required=True),
+password=dict(type='str', default=None, required=True),
+cluster_name=dict(type='str', default=None, required=True),
+state=dict(type='str', default=None, required=True,
+   choices=['started', 'stopped', 'deleted']),
+service_name=dict(type='str', required=False),
+component_name=dict(type='str', default=None, required=False),
+component_host=dict(type='str', 

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104654898
  
--- Diff: metron-deployment/extra_modules/ambari_service_state.py ---
@@ -0,0 +1,352 @@
+#!/usr/bin/python
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+DOCUMENTATION = '''
+---
+module: ambari_service_state
+version_added: "2.1"
+author: Apache Metron (Incubating : 
https://github.com/apache/incubator-metron )
+short_description: Start/Stop/Change Service or Component State
+description:
+- Start/Stop/Change Service or Component State
+options:
+  host:
+description:
+  The hostname for the ambari web server
+  port:
+description:
+  The port for the ambari web server
+  username:
+description:
+  The username for the ambari web server
+  password:
+description:
+  The name of the cluster in web server
+required: yes
+  cluster_name:
+description:
+  The name of the cluster in ambari
+required: yes
+  service_name:
+description:
+  The name of the service to alter
+required: no
+  component_name:
+description:
+  The name of the component to alter
+required: no
+  component_host:
+description:
+  The host running the targeted component. Required when 
component_name is used.
+required: no
+  state:
+description:
+  The desired service/component state.
+  wait_for_complete:
+description:
+  Whether to wait for the request to complete before returning. 
Default is False.
+required: no
+  requirements: [ 'requests']
+'''
+
+EXAMPLES = '''
+# must use full relative path to any files in stored in 
roles/role_name/files/
+- name: Create a new ambari cluster
+ambari_cluster_state:
+  host: localhost
+  port: 8080
+  username: admin
+  password: admin
+  cluster_name: my_cluster
+  cluster_state: present
+  blueprint_var: roles/my_role/files/blueprint.yml
+  blueprint_name: hadoop
+  wait_for_complete: True
+- name: Start the ambari cluster
+  ambari_cluster_state:
+host: localhost
+port: 8080
+username: admin
+password: admin
+cluster_name: my_cluster
+cluster_state: started
+wait_for_complete: True
+- name: Stop the ambari cluster
+  ambari_cluster_state:
+host: localhost
+port: 8080
+username: admin
+password: admin
+cluster_name: my_cluster
+cluster_state: stopped
+wait_for_complete: True
+- name: Delete the ambari cluster
+  ambari_cluster_state:
+host: localhost
+port: 8080
+username: admin
+password: admin
+cluster_name: my_cluster
+cluster_state: absent
+'''
+
+RETURN = '''
+results:
+description: The content of the requests object returned from the 
RESTful call
+returned: success
+type: string
+'''
+
+__author__ = 'apachemetron'
+
+import json
+
+try:
+import requests
+except ImportError:
+REQUESTS_FOUND = False
+else:
+REQUESTS_FOUND = True
+
+
+def main():
+
+argument_spec = dict(
+host=dict(type='str', default=None, required=True),
+port=dict(type='int', default=None, required=True),
+username=dict(type='str', default=None, required=True),
+password=dict(type='str', default=None, required=True),
+cluster_name=dict(type='str', default=None, required=True),
+state=dict(type='str', default=None, required=True,
+   choices=['started', 'stopped', 'deleted']),
+service_name=dict(type='str', required=False),
+component_name=dict(type='str', default=None, required=False),
+component_host=dict(type='str', 

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104658438
  
--- Diff: metron-deployment/roles/ambari_config/vars/single_node_vm.yml ---
@@ -80,10 +88,32 @@ configurations:
   - kafka-broker:
   log.dirs: '{{ kafka_log_dirs }}'
   delete.topic.enable: "true"
+  - metron-env:
+  parsers: "bro,snort"
+  - elastic-site:
+  index_number_of_shards: 1
+  index_number_of_replicas: 0
+  zen_discovery_ping_unicast_hosts: "{{ groups.search | join(',') }}"
+  gateway_recover_after_data_nodes: 1
+  network_host: "_lo_,_eth0_,_eth1_"
+  masters_also_are_datanodes: "1"
--- End diff --

wasn't this a boolean earlier? Should this be true?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104655538
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml
 ---
@@ -27,6 +27,14 @@
 Cluster name identifies your cluster
 
 
+masters_also_are_datanodes
+"false"
--- End diff --

Can we refactor things so that this is just `false`, rather than `"false"`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104661243
  
--- Diff: metron-deployment/roles/quick_dev/tasks/main.yml ---
@@ -15,23 +15,50 @@
 #  limitations under the License.
 #
 ---
-#
-# Workaround for Kafka not starting
-# Fire off async start followed by
-# Sync start -execution will pause until
-# final start completes.
-#
-- name: Start the ambari cluster - no wait
-  ambari_cluster_state:
+- name: Delete the Metron Components from Ambari
+  ambari_service_state:
 host: "{{ groups.ambari_master[0] }}"
 port: "{{ ambari_port }}"
 username: "{{ ambari_user }}"
 password: "{{ ambari_password }}"
 cluster_name: "{{ cluster_name }}"
-cluster_state: started
-wait_for_complete: False
+state: deleted
+component_name: "{{ item }}"
+component_host: "{{ inventory_hostname }}"
+  with_items:
+- METRON_ENRICHMENT_MASTER
+- METRON_INDEXING
+- METRON_PARSERS
+
+- name: Remove the Metron packages
+  package:
+name: "{{ item }}"
+state: absent
+  with_items:
+- metron-common
+- metron-data-management
+- metron-parsers
+- metron-enrichment
+- metron-indexing
+- metron-elasticsearch
+
+- name: Re-install the Metron Packages via Ambari
--- End diff --

Do we have any issues with the configured files still existing after this? 
I know RPMs don't like to touch files they didn't explicitly create, so I'm a 
little worried they still exist here.

I know guards were added (the `if(is_*_configured` earlier), but just want 
to call it out.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104660014
  
--- Diff: metron-deployment/roles/kibana/README.md ---
@@ -1,35 +0,0 @@
-Kibana 4
-
-
-This role installs Kibana along with the default Metron Dashboard.
-
-### FAQ
-
- How do I change Metron's default dashboard?
--- End diff --

Oh, hey, thanks! Maybe at the end of the deployment readme? I dunno, 
weirdly enough, I really struggled with a sensible place to put it. Co-located 
with the module seemed too buried (though, I hope you found the command line 
help, well, helpful) and way up in deployment seemed mis-located. I was about 
to put a wiki post together, but then there was some talk about not having 
one...
Anyway, maybe @ottobackwards has a suggestion, he pointed out the lack of 
documentation in the first place.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104653567
  
--- Diff: metron-deployment/roles/kibana/README.md ---
@@ -1,35 +0,0 @@
-Kibana 4
-
-
-This role installs Kibana along with the default Metron Dashboard.
-
-### FAQ
-
- How do I change Metron's default dashboard?
--- End diff --

If you know where you want it, I'm actually writing that up as part of 
METRON-745 (which needed to use that module).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104569239
  
--- Diff: metron-deployment/roles/metron-builder/tasks/main.yml ---
@@ -15,4 +15,7 @@
 #  limitations under the License.
 #
 ---
-hdp_repo_def: 
"http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.5.0.0/hdp.repo;
+- name: Build Deployment Artifacts
+  local_action: shell cd {{ metron_build_dir }} && mvn clean package 
-DskipTests -P HDP-2.5.0.0,mpack,build-rpms
--- End diff --

Yes, separate JIRA works for me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104565691
  
--- Diff: metron-deployment/roles/metron-builder/tasks/main.yml ---
@@ -15,4 +15,7 @@
 #  limitations under the License.
 #
 ---
-hdp_repo_def: 
"http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.5.0.0/hdp.repo;
+- name: Build Deployment Artifacts
+  local_action: shell cd {{ metron_build_dir }} && mvn clean package 
-DskipTests -P HDP-2.5.0.0,mpack,build-rpms
--- End diff --

I think I'd rather check in Maven and fail better earlier. What would you 
think about handling that as a standalone Jira?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104565422
  
--- Diff: metron-deployment/roles/quick_dev/tasks/main.yml ---
@@ -15,23 +15,50 @@
 #  limitations under the License.
 #
 ---
-#
-# Workaround for Kafka not starting
-# Fire off async start followed by
-# Sync start -execution will pause until
-# final start completes.
-#
-- name: Start the ambari cluster - no wait
-  ambari_cluster_state:
+- name: Delete the Metron Components from Ambari
--- End diff --

Exactly. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104558938
  
--- Diff: metron-deployment/roles/metron-builder/tasks/main.yml ---
@@ -15,4 +15,7 @@
 #  limitations under the License.
 #
 ---
-hdp_repo_def: 
"http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.5.0.0/hdp.repo;
+- name: Build Deployment Artifacts
+  local_action: shell cd {{ metron_build_dir }} && mvn clean package 
-DskipTests -P HDP-2.5.0.0,mpack,build-rpms
--- End diff --

Maybe something like...
```
- name: Build Deployment Artifacts
   local_action: shell cd {{ metron_build_dir }} && mvn clean package 
-DskipTests -P HDP-2.5.0.0,mpack,build-rpms
   become: false
   run_once: true
   ignore_errors: yes
   register: results

- fails: msg="Failed to build Metron... Is Docker Running?  "
   when: results.rc != 0
```

Or could we splits the Metron build into multiple steps?  First do the `mvn 
clean package`.  Then build the RPMS in a second step (the step that adds the 
Docker dependency) , so we know more concretely when the issue is likely to be 
that Docker not installed or not running?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104558496
  
--- Diff: metron-deployment/roles/metron-builder/tasks/main.yml ---
@@ -15,4 +15,7 @@
 #  limitations under the License.
 #
 ---
-hdp_repo_def: 
"http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.5.0.0/hdp.repo;
+- name: Build Deployment Artifacts
+  local_action: shell cd {{ metron_build_dir }} && mvn clean package 
-DskipTests -P HDP-2.5.0.0,mpack,build-rpms
--- End diff --

This is a "nice to have", but I noticed is that if you run "Full Dev" 
`vagrant up` without Docker running, it fails at this step with an awfully long 
and convoluted error message.  

```
...
ed(Launcher.java:289)", "\tat 
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)", 
"\tat 
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)",
 "\tat 
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)", 
"[INFO] 
", 
"[INFO] Reactor Summary:", "[INFO] ", "[INFO] Metron 
. SUCCESS [  0.327 s]", "[INFO] 
metron-analytics ... SUCCESS [  0.002 s]", 
"[INFO] metron-maas-common . SUCCESS [  5.415 
s]", "[INFO] metron-platform  SUCCESS [  
0.001 s]", "[INFO] metron-test-utilities .. SUCCESS 
[  3.629 s]", "[INFO] metron-integration-test  
SUCCESS [  7.044 s]", "[INFO] metron-maas-service 
 SUCCESS [  3.743 s]", "[INFO] 
 metron-common .. SUCCESS [ 18.669 s]", 
"[INFO] metron-statistics .. SUCCESS [ 13.954 
s]", "[INFO] metron-hbase ... SUCCESS [  
2.471 s]", "[INFO] metron-profiler-common . SUCCESS 
[  0.754 s]", "[INFO] metron-profiler-client . 
SUCCESS [ 15.322 s]", "[INFO] metron-profiler 
 SUCCESS [ 32.045 s]", "[INFO] 
metron-writer .. SUCCESS [ 12.181 s]", 
"[INFO] metron-enrichment .. SUCCESS [ 39.843 
s]", "[INFO] metron-indexing  SUCCESS [ 
12.021 s]", "[INFO] metron-solr  
SUCCESS [ 14.621 s]", "[INFO] metron-pcap 
 SUCCESS [  1.248 s]", "[INFO] 
metron-parsers . SUCCESS [ 40.486 s]", 
"[INFO] m
 etron-pcap-backend  SUCCESS [ 22.573 s]", 
"[INFO] metron-data-management . SUCCESS [ 45.768 
s]", "[INFO] metron-api . SUCCESS [ 
20.498 s]", "[INFO] metron-management .. 
SUCCESS [  4.632 s]", "[INFO] elasticsearch-shaded 
... SUCCESS [  3.916 s]", "[INFO] 
metron-elasticsearch ... SUCCESS [ 44.956 s]", 
"[INFO] metron-deployment .. SUCCESS [  0.000 
s]", "[INFO] metron-rpm . FAILURE [ 
10.066 s]", "[INFO] Metron Ambari Management Pack .. 
SKIPPED", "[INFO] metron-docker .. 
SKIPPED", "[INFO] 
", 
"[INFO] BUILD FAILURE", "[INFO] 
", 
"[INFO] To
 tal time: 06:16 min", "[INFO] Finished at: 2017-03-06T19:09:39-05:00", "[INFO] 
Final Memory: 162M/1993M", "[INFO] 
", 
"[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:exec 
(docker-build) on project metron-rpm: Command execution failed. Process exited 
with an error: 1 (Exit value: 1) -> [Help 1]", "[ERROR] ", "[ERROR] To see the 
full stack trace of the errors, re-run Maven with the -e switch.", "[ERROR] 
Re-run Maven using the -X switch to enable full debug logging.", "[ERROR] ", 
"[ERROR] For more information about the errors and possible solutions, please 
read the following articles:", "[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException;, 
"[ERROR] ", "[ERROR] After correcting the problems, you can resume the build 
with the command", "[ERROR]   mvn  -rf :metron-rpm"], "warnings": []}
```

I think this might cause some headaches, especially for new folks.  
Ideally, we could add some kind of more intelligible error message.



---
If your project is set up for it, you can reply to this email and 

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104553128
  
--- Diff: metron-deployment/roles/quick_dev/tasks/main.yml ---
@@ -15,23 +15,50 @@
 #  limitations under the License.
 #
 ---
-#
-# Workaround for Kafka not starting
-# Fire off async start followed by
-# Sync start -execution will pause until
-# final start completes.
-#
-- name: Start the ambari cluster - no wait
-  ambari_cluster_state:
+- name: Delete the Metron Components from Ambari
--- End diff --

Ah, ok.  So our Atlas image has Metron pre-installed via the MPack.  But 
this role then uninstalls and reinstalls it so that we get the latest, right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104546536
  
--- Diff: metron-deployment/packaging/packer-build/.gitignore ---
@@ -1,6 +1,6 @@
 .bundle/
 iso
-*.box

+builds/base-centos-6.7-2.1.20170303223924.git.33abe8cf13c347a2dfdece145a7b8c17f2a423c0_dirty.virtualbox.box
--- End diff --

Urm, no. I must have dumped my copy buffer by mistake while IntelliJ was 
open. It'll be gone in the next changeset. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104546225
  
--- Diff: metron-deployment/vagrant/full-dev-platform/Vagrantfile ---
@@ -53,7 +53,7 @@ hosts = [{
 Vagrant.configure(2) do |config|
 
   # all hosts built on centos 6
-  config.vm.box = "bento/centos-6.7"
+  config.vm.box = "metron/centos_base"
--- End diff --

Exactly that for exactly that reason. That and I can control the swap space 
size now. I get a much more reliable (not perfect, but better) full dev 
experience with this image. Something that I wanted to do for quite some time, 
but I finally got it done because I was having trouble differentiating between 
errors I had introduced and pre-existing errors. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104545807
  
--- Diff: metron-deployment/roles/quick_dev/tasks/main.yml ---
@@ -15,23 +15,50 @@
 #  limitations under the License.
 #
 ---
-#
-# Workaround for Kafka not starting
-# Fire off async start followed by
-# Sync start -execution will pause until
-# final start completes.
-#
-- name: Start the ambari cluster - no wait
-  ambari_cluster_state:
+- name: Delete the Metron Components from Ambari
--- End diff --

It bridges the gap between the quick-dev image and a fully functioning 
instance. To wit:

- Removes the stale Metron that ships with the image from Ambari
- Removes the stale Metron RPMs
- Re-installs fresh Metron packages
- Starts the cluster

We only require these activities on a quick-dev image. If you think of 
Ansible as declarative, it's not that weird. We apply the quick-dev role to the 
quick-dev host.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104544895
  
--- Diff: metron-deployment/roles/kibana/README.md ---
@@ -1,35 +0,0 @@
-Kibana 4
-
-
-This role installs Kibana along with the default Metron Dashboard.
-
-### FAQ
-
- How do I change Metron's default dashboard?
--- End diff --

There's a Python 
[module](https://github.com/apache/incubator-metron/tree/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/dashboard)
 that I wrote as part of the MPack that does the work there. I have an 
outstanding item to write up it's basic usage, but can't find a good place in 
the code. Ideas?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104539908
  
--- Diff: metron-deployment/roles/kafka-broker/tasks/main.yml ---
@@ -1,41 +0,0 @@
-#
-#  Licensed to the Apache Software Foundation (ASF) under one or more
-#  contributor license agreements.  See the NOTICE file distributed with
-#  this work for additional information regarding copyright ownership.
-#  The ASF licenses this file to You under the Apache License, Version 2.0
-#  (the "License"); you may not use this file except in compliance with
-#  the License.  You may obtain a copy of the License at
-#
-#  http://www.apache.org/licenses/LICENSE-2.0
-#
-#  Unless required by applicable law or agreed to in writing, software
-#  distributed under the License is distributed on an "AS IS" BASIS,
-#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-#  See the License for the specific language governing permissions and
-#  limitations under the License.
-#

-- name: Retrieve HDP repository definition
-  get_url:
-url: "{{ hdp_repo_def }}"
-dest: /etc/yum.repos.d/hdp.repo
-mode: 0644
-
-- name: Install kafka
-  yum: name={{item}}
-  with_items:
-- java-1.8.0-openjdk
-- kafka
-- zookeeper-server
-
-- name: Create pid directories
-  file: path={{ item }} state=directory mode=0755
-  with_items:
-- /var/run/zookeeper
-- /var/run/kafka
-
-- name: Start zookeeper
-  shell: /usr/hdp/current/zookeeper-server/bin/zookeeper-server start
-
-- name: Start kafka
--- End diff --

The `kafka_broker` role is used by the `fastcapa-test-platform`.  Removing 
it will break that environment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104540418
  
--- Diff: metron-deployment/roles/kibana/README.md ---
@@ -1,35 +0,0 @@
-Kibana 4
-
-
-This role installs Kibana along with the default Metron Dashboard.
-
-### FAQ
-
- How do I change Metron's default dashboard?
--- End diff --

How does the load of the Kibana Dashboard work now?  Do we not use 
`elasticdump` any longer?  Bravo, if you found a better way.

This README was intended to help describe how to make changes to the Metron 
Kibana Dashboard and get those changes into source control.  I just want to 
make sure this same sort of thing is either obvious now or described somewhere 
else.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104542365
  
--- Diff: metron-deployment/roles/quick_dev/tasks/main.yml ---
@@ -15,23 +15,50 @@
 #  limitations under the License.
 #
 ---
-#
-# Workaround for Kafka not starting
-# Fire off async start followed by
-# Sync start -execution will pause until
-# final start completes.
-#
-- name: Start the ambari cluster - no wait
-  ambari_cluster_state:
+- name: Delete the Metron Components from Ambari
--- End diff --

What does the `quick-dev` role do?  Curious why we have a role specific to 
one of our environments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104543437
  
--- Diff: metron-deployment/vagrant/full-dev-platform/Vagrantfile ---
@@ -53,7 +53,7 @@ hosts = [{
 Vagrant.configure(2) do |config|
 
   # all hosts built on centos 6
-  config.vm.box = "bento/centos-6.7"
+  config.vm.box = "metron/centos_base"
--- End diff --

How did our Atlas environment change?  It seems like we now create our one 
CentOS image?  

Seems like a good idea being that Bento's image could and did change 
without warning.  Do I build the CentOS image with the rest of the 
`packer-build` stuff?  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r104537561
  
--- Diff: metron-deployment/packaging/packer-build/.gitignore ---
@@ -1,6 +1,6 @@
 .bundle/
 iso
-*.box

+builds/base-centos-6.7-2.1.20170303223924.git.33abe8cf13c347a2dfdece145a7b8c17f2a423c0_dirty.virtualbox.box
--- End diff --

Did you intend to make this change?  Seems oddly specific. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r103241052
  
--- Diff: metron-deployment/roles/java_jdk/defaults/main.yml ---
@@ -15,4 +15,4 @@
 #  limitations under the License.
 #
 ---
--- End diff --

I used Ambari's default so I don't have to worry about any introduced 
non-compatibility.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r103240842
  
--- Diff: metron-deployment/roles/kibana/README.md ---
@@ -1,35 +0,0 @@
-Kibana 4
--- End diff --

Ah, it is, but not well. I'll make sure I get it as part of this effort.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r103239678
  
--- Diff: metron-deployment/roles/java_jdk/defaults/main.yml ---
@@ -15,4 +15,4 @@
 #  limitations under the License.
 #
 ---
--- End diff --

Is there a reason not to go to the latest ( _121 ) if we are going to 
update?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r103238671
  
--- Diff: metron-deployment/roles/kibana/README.md ---
@@ -1,35 +0,0 @@
-Kibana 4
--- End diff --

The file I commented was the readme that documented how to add or modify 
the kibana page template


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

Github user dlyle65535 commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r103235875
  
--- Diff: metron-deployment/roles/kibana/README.md ---
@@ -1,35 +0,0 @@
-Kibana 4
--- End diff --

Is what documented? I just removed the role as it's deprecated with the 
MPack.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/436#discussion_r103235131
  
--- Diff: metron-deployment/roles/kibana/README.md ---
@@ -1,35 +0,0 @@
-Kibana 4
--- End diff --

Is this documented anywhere else?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...

[dlyle65535]

GitHub user dlyle65535 opened a pull request:

https://github.com/apache/incubator-metron/pull/436

METRON-671: Refactor existing Ansible deployment to use Ambari MPack

**WORK IN PROGRESS - DO NOT MERGE**

This is the first set of changes to enable Ansible installation to use the 
Ambari MPack. It currently works (in my environment) with full-dev using 
sensor-stubs. It will not work with Quick-Dev or EC2 at this time.

It's (well past - sorry) starting to get large, so I wanted to push it out 
for feedback while I'm working issues with the distributed install.

Some points of interest:
- As discussed, components (ES, Kibana, Metron Topologies) that are managed 
by Ambari are removed from Monit
- I had added a notion of 'required configurations' to the blueprint and 
ambari_cluster_state script so blueprint verification would pass.
- I've refactored our Ansible host groupings (see 
inventory/full-dev-platform/hosts). The Metron group is now the host that we 
use as the Metron services master.
- I cleaned up the very minimum number of issues that @mattf-horton found 
in [METRON-609](https://issues.apache.org/jira/browse/METRON-609) required to 
make single-node deployment work with the MPack. It's my intention to work to 
get his changes incorporated on top of the final version of this changeset.
- I removed the Solr role from Ansible. My current intention is to open a 
Jira to add a Solr Indexing component to the MPack.

Immediate next steps:

- [ ] Get the EC2 distributed install working.
- [ ] Adjust readmes and docs where required.
- [ ] Merge in Matt's changes and make required PRs on his branch (if any).
- [ ] Document test plan

I also hope to receive and respond to feedback.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dlyle65535/incubator-metron METRON-671

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/436.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #436


commit 01cd4216754861c7eaf58c04472fb2a8125a6f28
Author: David Lyle 
Date:   2017-01-23T20:47:14Z

METRON-671: Refactor existing Ansible deployment to use Ambari MPack




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---