[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308816#comment-16308816 ] ASF GitHub Bot commented on METRON-1362: Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159330127 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,134 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services +This project contains tools for building, packaging, and deploying Apache Metron. Please refer to the following sections for more information on how to get Apache Metron running in your environment. -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites -- A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](#rpms) for further information. -- [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the Management UI host - -### Building Management Pack -From `metron-deployment` run -``` -mvn clean package -``` - -A tar.gz that can be used with Ambari can be found at `metron-deployment/packaging/ambari/metron-mpack/target/` - -### Installing Management Pack -Before installing the mpack, update Storm's topology.classpath in Ambari to include '/etc/hbase/conf:/etc/hadoop/conf'. Restart Storm service. - -Place the mpack's tar.gz onto the node running Ambari Server. From the command line on this node, run -``` -ambari-server install-mpack --mpack= --verbose -``` - -This will make the services available in Ambari in the same manner as any services in a stack, e.g. through Add Services or during cluster install. -The Indexing / Parsers/ Enrichment masters should be colocated with a Kafka Broker (to create topics) and HBase client (to create the enrichment and theatintel tables). -This colocation is currently not enforced by Ambari, and should be managed by either a Service or Stack advisor as an enhancement. - -Several configuration parameters will need to be filled in, and should be pretty self explanatory (primarily a couple of Elasticsearch configs, and the Storm REST URL). Examples are provided in the descriptions on Ambari. -Notably, the
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308801#comment-16308801 ] ASF GitHub Bot commented on METRON-1362: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159327372 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,134 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services +This project contains tools for building, packaging, and deploying Apache Metron. Please refer to the following sections for more information on how to get Apache Metron running in your environment. -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites -- A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](#rpms) for further information. -- [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the Management UI host - -### Building Management Pack -From `metron-deployment` run -``` -mvn clean package -``` - -A tar.gz that can be used with Ambari can be found at `metron-deployment/packaging/ambari/metron-mpack/target/` - -### Installing Management Pack -Before installing the mpack, update Storm's topology.classpath in Ambari to include '/etc/hbase/conf:/etc/hadoop/conf'. Restart Storm service. - -Place the mpack's tar.gz onto the node running Ambari Server. From the command line on this node, run -``` -ambari-server install-mpack --mpack= --verbose -``` - -This will make the services available in Ambari in the same manner as any services in a stack, e.g. through Add Services or during cluster install. -The Indexing / Parsers/ Enrichment masters should be colocated with a Kafka Broker (to create topics) and HBase client (to create the enrichment and theatintel tables). -This colocation is currently not enforced by Ambari, and should be managed by either a Service or Stack advisor as an enhancement. - -Several configuration parameters will need to be filled in, and should be pretty self explanatory (primarily a couple of Elasticsearch configs, and the Storm REST URL). Examples are provided in the descriptions on Ambari. -Notably, the
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308800#comment-16308800 ] ASF GitHub Bot commented on METRON-1362: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159327218 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,134 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services +This project contains tools for building, packaging, and deploying Apache Metron. Please refer to the following sections for more information on how to get Apache Metron running in your environment. -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites -- A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](#rpms) for further information. -- [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the Management UI host - -### Building Management Pack -From `metron-deployment` run -``` -mvn clean package -``` - -A tar.gz that can be used with Ambari can be found at `metron-deployment/packaging/ambari/metron-mpack/target/` - -### Installing Management Pack -Before installing the mpack, update Storm's topology.classpath in Ambari to include '/etc/hbase/conf:/etc/hadoop/conf'. Restart Storm service. - -Place the mpack's tar.gz onto the node running Ambari Server. From the command line on this node, run -``` -ambari-server install-mpack --mpack= --verbose -``` - -This will make the services available in Ambari in the same manner as any services in a stack, e.g. through Add Services or during cluster install. -The Indexing / Parsers/ Enrichment masters should be colocated with a Kafka Broker (to create topics) and HBase client (to create the enrichment and theatintel tables). -This colocation is currently not enforced by Ambari, and should be managed by either a Service or Stack advisor as an enhancement. - -Several configuration parameters will need to be filled in, and should be pretty self explanatory (primarily a couple of Elasticsearch configs, and the Storm REST URL). Examples are provided in the descriptions on Ambari. -Notably, the
[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections
[ https://issues.apache.org/jira/browse/METRON-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308776#comment-16308776 ] ASF GitHub Bot commented on METRON-1302: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/831 I ran this up in full dev and everything worked as advertised. I only noticed a couple minor issues and left comments for those. I also am a little confused by the ra/batch vs es/hdfs issue. I still see places (Ambari config parameter names, Ambari MPack scripts, Flux file paths, Flux properties, Storm topology names, etc) where the prefixes are ra/batch and not es/hdfs. Reading through the PR comments I'm still not clear on what approach we decided on but I think consistency would be good. > Split up Indexing Topology into batch and random access sections > > > Key: METRON-1302 > URL: https://issues.apache.org/jira/browse/METRON-1302 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella > > Currently we have the indexing topology handle writing to both random access > indices (e.g. elasticsearch) as well as batch write indices (e.g. hdfs). We > should split these up and configure them separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308774#comment-16308774 ] ASF GitHub Bot commented on METRON-1362: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159324176 --- Diff: metron-deployment/packaging/ambari/metron-mpack/README.md --- @@ -0,0 +1,94 @@ + + +This provides a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, management and monitoring of Metron on clusters of any size. + +This allows you to easily install Metron using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + +### Prerequisites + +* Ambari 2.4.2+ + +* Installable Metron packages (either RPMs or DEBs) located in a repository on each host at `/localrepo`. + +* A [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the host running the Management and Alarm UI. + +### Quick Start + +1. Build the Metron MPack. Execute the following command from the project's root directory. +``` +mvn clean package -Pmpack -DskipTests +``` + +1. This results in the Mpack being produced at the following location. +``` + metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-x.y.z.0.tar.gz +``` + +1. Copy the tarball to the host where Ambari Server is installed. + +1. Ensure that Ambari Server is stopped. + +1. Install the MPack. +``` +ambari-server install-mpack --mpack=metron_mpack-x.y.z.0.tar.gz --verbose +``` + +1. Metron swill now be available as an installable service within Ambari. + +### Installation Notes + +The MPack will make all Metron services available in Ambari in the same manner as any other services in a stack. These can be installed using Ambari's user interface using "Add Services" or during an initial cluster install. + + Co-Location + +1. The Parsers, Enrichment, Indexing, and Profiler masters should be colocated on a host with a Kafka Broker. This is necessary so that the correct Kafka topics can be created. + +1. The Enrichment and Profiler masters should be colocated on a host with an HBase client. This is necessary so that the Enrichment, Threat Intel, and Profile tables can be created. + +This colocation is currently not enforced by Ambari and should be managed by either a Service or Stack advisor as an enhancement. + + Kerberization + +The MPack allows Metron to be automatically kerberized in two different ways. +* Metron can be installed on a non-kerberized cluster and then the entire cluster can be kerberized using Ambari. +* Alternatively, Metron can be installed on top of an already kerberized cluster. + +Using the MPack is preferred, but instructions for manually Kerberizing a cluster with Metron can be found at [Kerberos-manual-setup.md](../../Kerberos-manual-setup.md). + + Zeppelin Import + +A custom action is available in Ambari to import Zeppelin dashboards. See the [metron-indexing documentation](../metron-platform/metron-indexing) for more information. + + Offline Installation + +There is only one point during installation that reaches out to the internet. That is necessary to retrieve the GeoIP database. --- End diff -- I have had a hard time following the installation notes around the Mpack, so I definitely might have missed something there. I just went back and added some additional stuff. Hopefully that is everything that is still relevant. > Improve Metron Deployment README > > > Key: METRON-1362 > URL: https://issues.apache.org/jira/browse/METRON-1362 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > The README and docs under metron-deployment/ are rather confusing and > containing outdated information. I'd like to get that cleaned up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308767#comment-16308767 ] ASF GitHub Bot commented on METRON-1362: Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159322553 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,134 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services +This project contains tools for building, packaging, and deploying Apache Metron. Please refer to the following sections for more information on how to get Apache Metron running in your environment. -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites -- A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](#rpms) for further information. -- [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the Management UI host - -### Building Management Pack -From `metron-deployment` run -``` -mvn clean package -``` - -A tar.gz that can be used with Ambari can be found at `metron-deployment/packaging/ambari/metron-mpack/target/` - -### Installing Management Pack -Before installing the mpack, update Storm's topology.classpath in Ambari to include '/etc/hbase/conf:/etc/hadoop/conf'. Restart Storm service. - -Place the mpack's tar.gz onto the node running Ambari Server. From the command line on this node, run -``` -ambari-server install-mpack --mpack= --verbose -``` - -This will make the services available in Ambari in the same manner as any services in a stack, e.g. through Add Services or during cluster install. -The Indexing / Parsers/ Enrichment masters should be colocated with a Kafka Broker (to create topics) and HBase client (to create the enrichment and theatintel tables). -This colocation is currently not enforced by Ambari, and should be managed by either a Service or Stack advisor as an enhancement. - -Several configuration parameters will need to be filled in, and should be pretty self explanatory (primarily a couple of Elasticsearch configs, and the Storm REST URL). Examples are provided in the descriptions on Ambari. -Notably, the
[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections
[ https://issues.apache.org/jira/browse/METRON-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308766#comment-16308766 ] ASF GitHub Bot commented on METRON-1302: Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/831#discussion_r159322924 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/hdfs.properties.j2 --- @@ -0,0 +1,44 @@ +# Licensed to the Apache Software Foundation (ASF) under one --- End diff -- Should this file be git ignored since it is copied in at build time? > Split up Indexing Topology into batch and random access sections > > > Key: METRON-1302 > URL: https://issues.apache.org/jira/browse/METRON-1302 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella > > Currently we have the indexing topology handle writing to both random access > indices (e.g. elasticsearch) as well as batch write indices (e.g. hdfs). We > should split these up and configure them separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1302) Split up Indexing Topology into batch and random access sections
[
https://issues.apache.org/jira/browse/METRON-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308751#comment-16308751
]
ASF GitHub Bot commented on METRON-1302:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/831#discussion_r159321881
--- Diff:
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/integration/IndexingIntegrationTest.java
---
@@ -197,9 +140,7 @@ public void test() throws Exception {
//assert that our input docs are equivalent to the output docs,
converting the input docs keys based
// on the field name converter
assertInputDocsMatchOutputs(inputDocs, docs,
getFieldNameConverter());
- assertInputDocsMatchOutputs(inputDocs, readDocsFromDisk(hdfsDir), x
-> x);
-} catch(Throwable e) {
- e.printStackTrace();
+ //assertInputDocsMatchOutputs(inputDocs, readDocsFromDisk(hdfsDir),
x -> x);
--- End diff --
Is this comment intentional?
> Split up Indexing Topology into batch and random access sections
>
>
> Key: METRON-1302
> URL: https://issues.apache.org/jira/browse/METRON-1302
> Project: Metron
> Issue Type: Improvement
>Reporter: Casey Stella
>
> Currently we have the indexing topology handle writing to both random access
> indices (e.g. elasticsearch) as well as batch write indices (e.g. hdfs). We
> should split these up and configure them separately.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308723#comment-16308723 ] ASF GitHub Bot commented on METRON-1362: Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159317149 --- Diff: metron-deployment/packaging/ambari/metron-mpack/README.md --- @@ -0,0 +1,94 @@ + + +This provides a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, management and monitoring of Metron on clusters of any size. + +This allows you to easily install Metron using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + +### Prerequisites + +* Ambari 2.4.2+ + +* Installable Metron packages (either RPMs or DEBs) located in a repository on each host at `/localrepo`. + +* A [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the host running the Management and Alarm UI. + +### Quick Start + +1. Build the Metron MPack. Execute the following command from the project's root directory. +``` +mvn clean package -Pmpack -DskipTests +``` + +1. This results in the Mpack being produced at the following location. +``` + metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-x.y.z.0.tar.gz +``` + +1. Copy the tarball to the host where Ambari Server is installed. + +1. Ensure that Ambari Server is stopped. + +1. Install the MPack. +``` +ambari-server install-mpack --mpack=metron_mpack-x.y.z.0.tar.gz --verbose +``` + +1. Metron swill now be available as an installable service within Ambari. + +### Installation Notes + +The MPack will make all Metron services available in Ambari in the same manner as any other services in a stack. These can be installed using Ambari's user interface using "Add Services" or during an initial cluster install. + + Co-Location + +1. The Parsers, Enrichment, Indexing, and Profiler masters should be colocated on a host with a Kafka Broker. This is necessary so that the correct Kafka topics can be created. + +1. The Enrichment and Profiler masters should be colocated on a host with an HBase client. This is necessary so that the Enrichment, Threat Intel, and Profile tables can be created. + +This colocation is currently not enforced by Ambari and should be managed by either a Service or Stack advisor as an enhancement. + + Kerberization + +The MPack allows Metron to be automatically kerberized in two different ways. +* Metron can be installed on a non-kerberized cluster and then the entire cluster can be kerberized using Ambari. +* Alternatively, Metron can be installed on top of an already kerberized cluster. + +Using the MPack is preferred, but instructions for manually Kerberizing a cluster with Metron can be found at [Kerberos-manual-setup.md](../../Kerberos-manual-setup.md). + + Zeppelin Import + +A custom action is available in Ambari to import Zeppelin dashboards. See the [metron-indexing documentation](../metron-platform/metron-indexing) for more information. + + Offline Installation + +There is only one point during installation that reaches out to the internet. That is necessary to retrieve the GeoIP database. --- End diff -- Sidenote, that should just be two slashes there (`file://`), since the extra `/` is the root dir, iirc. > Improve Metron Deployment README > > > Key: METRON-1362 > URL: https://issues.apache.org/jira/browse/METRON-1362 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > The README and docs under metron-deployment/ are rather confusing and > containing outdated information. I'd like to get that cleaned up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308668#comment-16308668 ] ASF GitHub Bot commented on METRON-1362: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/869 +1 Thanks > Improve Metron Deployment README > > > Key: METRON-1362 > URL: https://issues.apache.org/jira/browse/METRON-1362 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > The README and docs under metron-deployment/ are rather confusing and > containing outdated information. I'd like to get that cleaned up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1231) Separate Sensor name and topic in the Management UI
[ https://issues.apache.org/jira/browse/METRON-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308517#comment-16308517 ] ASF GitHub Bot commented on METRON-1231: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/786 @justinleet this should be ready for review now. > Separate Sensor name and topic in the Management UI > --- > > Key: METRON-1231 > URL: https://issues.apache.org/jira/browse/METRON-1231 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman > > Currently the Management UI (and REST app) assumes the sensor name == sensor > topic which is false. These should be handles separately and the UI should > expose a way to edit both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308459#comment-16308459 ] ASF GitHub Bot commented on METRON-1362: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159284348 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,132 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning,
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308457#comment-16308457 ] ASF GitHub Bot commented on METRON-1362: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159284273 --- Diff: metron-deployment/README.md --- @@ -1,175 +1,129 @@ -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308439#comment-16308439 ] ASF GitHub Bot commented on METRON-1362: Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159280034 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,132 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning,
[jira] [Commented] (METRON-1384) Increment master version number to 0.4.3 for on-going development
[ https://issues.apache.org/jira/browse/METRON-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308435#comment-16308435 ] ASF GitHub Bot commented on METRON-1384: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/885 > Increment master version number to 0.4.3 for on-going development > - > > Key: METRON-1384 > URL: https://issues.apache.org/jira/browse/METRON-1384 > Project: Metron > Issue Type: Bug >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 0.4.2 > > > ...now that 0.4.2 is being released to public. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308412#comment-16308412 ] ASF GitHub Bot commented on METRON-1362: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/869 Ok, @ottobackwards. I'll take a hack at that. > Improve Metron Deployment README > > > Key: METRON-1362 > URL: https://issues.apache.org/jira/browse/METRON-1362 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > The README and docs under metron-deployment/ are rather confusing and > containing outdated information. I'd like to get that cleaned up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1385) Missing "properties" in index template causes ElasticsearchColumnMetadataDao.getColumnMetadata to fail
[
https://issues.apache.org/jira/browse/METRON-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308407#comment-16308407
]
ASF GitHub Bot commented on METRON-1385:
GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/886
METRON-1385: Missing "properties" in index template causes
ElasticsearchColumnMetadataDao.getColumnMetadata to fail
## Contributor Comments
A bug was recently discovered that causes a NPE when calling
ElasticsearchColumnMetadataDao.getColumnMetadata to get ES column types. It
can be recreated by adding a template to ES that matches all indices and also
contains a doc type mapping without the "properties" property. For example:
```
curl -XPUT 'http://node1:9200/_template/default_string_template' -d '
{
"template": "*",
"mappings" : {
"bro_type": {
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "text"
}
}
}
]
}
}
}
```
A NPE should now happen when calling the REST endpoint to get column
metadata for bro.
This PR fixes that bug by first checking to see if the "properties"
property exists and continuing on if it does not. I also added an additional
mapping in ElasticsearchSearchIntegrationTest that mimics this condition.
Happy to remove it if people feel it's unnecessary.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron METRON-1385
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/886.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #886
commit 8905d4d21926ac4e518d14fb0b4b44fbf4dea158
Author: merrimanr
Date: 2018-01-02T16:22:12Z
initial commit
> Missing "properties" in index template causes
>
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308402#comment-16308402 ] ASF GitHub Bot commented on METRON-939: --- Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r159266301 --- Diff: metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.md --- @@ -441,7 +441,7 @@ Client - Kibana: * Set "kibana_es_url" to `http://:9200`. "replace_with_elasticsearch_master_hostname" is the IP of the node where you assigned ElasticSearch Master on the Assign Master tab. -* Change kibana_default_application to "dashboard/Metron-Dashboard" +* Change kibana_default_application to "dashboard/AV-YpDmwdXwc6Ua9Muh9" --- End diff -- Is it possible to provide an easily identifiable name, or are we stuck with the "AV-..." ugliness? > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > Labels: backwards-incompatible > Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - > Kibana.pdf > > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308401#comment-16308401
]
ASF GitHub Bot commented on METRON-939:
---
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r159273549
--- Diff: pom.xml ---
@@ -159,7 +159,7 @@
${global_surefire_version}
-@{argLine} -Xmx2048m
+-Xmx2048m
--- End diff --
This was in for code coverage via JaCoCo. It basically overrides the empty
argline from above that got deleted. Was it causing problems with running
things with it there?
> Upgrade ElasticSearch and Kibana
>
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
> Issue Type: Improvement
>Reporter: Jon Zeolla
>Assignee: Michael Miklavcic
> Labels: backwards-incompatible
> Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard -
> Kibana.pdf
>
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among
> other benefits, this allows us to use periods in field names
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been
> available as of 5.0 and 2.4, and the ability to index an IPv6 address
> properly
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308368#comment-16308368 ] ASF GitHub Bot commented on METRON-1362: Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159268214 --- Diff: metron-deployment/README.md --- @@ -1,175 +1,129 @@ -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308366#comment-16308366 ] ASF GitHub Bot commented on METRON-1362: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/869 https://github.com/apache/metron/tree/master/metron-deployment/packaging/docker/ansible-docker > Improve Metron Deployment README > > > Key: METRON-1362 > URL: https://issues.apache.org/jira/browse/METRON-1362 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > The README and docs under metron-deployment/ are rather confusing and > containing outdated information. I'd like to get that cleaned up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308364#comment-16308364 ] ASF GitHub Bot commented on METRON-1362: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/869 The ansible-docker image can be used for this. > Improve Metron Deployment README > > > Key: METRON-1362 > URL: https://issues.apache.org/jira/browse/METRON-1362 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > The README and docs under metron-deployment/ are rather confusing and > containing outdated information. I'd like to get that cleaned up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308356#comment-16308356 ] ASF GitHub Bot commented on METRON-1362: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159266396 --- Diff: metron-deployment/README.md --- @@ -1,175 +1,129 @@ -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308354#comment-16308354 ] ASF GitHub Bot commented on METRON-939: --- Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r159265938 --- Diff: metron-deployment/README.md --- @@ -1,3 +1,16 @@ +# Metron Deployment --- End diff -- As a heads up, #883 is in now, so this will have to be taken care of when you merge master to deconflict. > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Michael Miklavcic > Labels: backwards-incompatible > Attachments: Metron-Dashboard - Kibana.pdf, Metron-Error-Dashboard - > Kibana.pdf > > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1381) Add Apache license to MD files and remove the Rat exclusion
[ https://issues.apache.org/jira/browse/METRON-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308333#comment-16308333 ] ASF GitHub Bot commented on METRON-1381: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/883 > Add Apache license to MD files and remove the Rat exclusion > --- > > Key: METRON-1381 > URL: https://issues.apache.org/jira/browse/METRON-1381 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet > > We should be adding the Apache license in an HTML comment () on our > Markdown files. This allows the header to still be in the source, but hides > it from the rendering. I validated that this works on both the GitHub > rendering and the site-book rendering. > This came up in a discuss thread awhile ago, but it looks like it slipped > through the cracks afterwards. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1381) Add Apache license to MD files and remove the Rat exclusion
[ https://issues.apache.org/jira/browse/METRON-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308329#comment-16308329 ] ASF GitHub Bot commented on METRON-1381: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/883 +1 > Add Apache license to MD files and remove the Rat exclusion > --- > > Key: METRON-1381 > URL: https://issues.apache.org/jira/browse/METRON-1381 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet > > We should be adding the Apache license in an HTML comment () on our > Markdown files. This allows the header to still be in the source, but hides > it from the rendering. I validated that this works on both the GitHub > rendering and the site-book rendering. > This came up in a discuss thread awhile ago, but it looks like it slipped > through the cracks afterwards. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308321#comment-16308321 ] ASF GitHub Bot commented on METRON-1362: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159261149 --- Diff: metron-deployment/README.md --- @@ -1,175 +1,129 @@ -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308317#comment-16308317 ] ASF GitHub Bot commented on METRON-1362: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/869 Do those instructions exist somewhere before this PR ? That sounds like new functionality or am I misunderstanding your request? > Improve Metron Deployment README > > > Key: METRON-1362 > URL: https://issues.apache.org/jira/browse/METRON-1362 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > The README and docs under metron-deployment/ are rather confusing and > containing outdated information. I'd like to get that cleaned up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1384) Increment master version number to 0.4.3 for on-going development
[ https://issues.apache.org/jira/browse/METRON-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308314#comment-16308314 ] ASF GitHub Bot commented on METRON-1384: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/885 I'll take care of getting this merged now. > Increment master version number to 0.4.3 for on-going development > - > > Key: METRON-1384 > URL: https://issues.apache.org/jira/browse/METRON-1384 > Project: Metron > Issue Type: Bug >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 0.4.2 > > > ...now that 0.4.2 is being released to public. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308309#comment-16308309 ] ASF GitHub Bot commented on METRON-1362: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/869 Can we have a section on `How dow I build everything with docker` with instructions for the ansible docker container? > Improve Metron Deployment README > > > Key: METRON-1362 > URL: https://issues.apache.org/jira/browse/METRON-1362 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > The README and docs under metron-deployment/ are rather confusing and > containing outdated information. I'd like to get that cleaned up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (METRON-1385) Missing "properties" in index template causes ElasticsearchColumnMetadataDao.getColumnMetadata to fail
Ryan Merriman created METRON-1385: - Summary: Missing "properties" in index template causes ElasticsearchColumnMetadataDao.getColumnMetadata to fail Key: METRON-1385 URL: https://issues.apache.org/jira/browse/METRON-1385 Project: Metron Issue Type: Bug Reporter: Ryan Merriman Assignee: Ryan Merriman It is possible to add an ES mapping to an index that does not have a "properties" property. This causes a NPE because the ElasticsearchColumnMetadataDao.getColumnMetadata method expects it to be there. It should safely skip the mapping instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1384) Increment master version number to 0.4.3 for on-going development
[ https://issues.apache.org/jira/browse/METRON-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308276#comment-16308276 ] ASF GitHub Bot commented on METRON-1384: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/885 I'm +1 by inspection, since Nick built the code base. Thanks for taking care of this! > Increment master version number to 0.4.3 for on-going development > - > > Key: METRON-1384 > URL: https://issues.apache.org/jira/browse/METRON-1384 > Project: Metron > Issue Type: Bug >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 0.4.2 > > > ...now that 0.4.2 is being released to public. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1384) Increment master version number to 0.4.3 for on-going development
[ https://issues.apache.org/jira/browse/METRON-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308254#comment-16308254 ] ASF GitHub Bot commented on METRON-1384: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/885 +1 Thanks, Matt! Manually reviewed the changes and built the code base. All remaining references to 0.4.2 are part of step-by-step instructions where the version does not really need changed. > Increment master version number to 0.4.3 for on-going development > - > > Key: METRON-1384 > URL: https://issues.apache.org/jira/browse/METRON-1384 > Project: Metron > Issue Type: Bug >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 0.4.2 > > > ...now that 0.4.2 is being released to public. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308207#comment-16308207 ] ASF GitHub Bot commented on METRON-1362: Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159246961 --- Diff: metron-deployment/README.md --- @@ -1,175 +1,129 @@ -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308166#comment-16308166 ] ASF GitHub Bot commented on METRON-1362: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/869 @justinleet Is that a +1 then? @ottobackwards What are your thoughts on this PR? Is this an incremental improvement that we can merge? > Improve Metron Deployment README > > > Key: METRON-1362 > URL: https://issues.apache.org/jira/browse/METRON-1362 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > The README and docs under metron-deployment/ are rather confusing and > containing outdated information. I'd like to get that cleaned up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1340) Improve e2e tests for metron alerts
[ https://issues.apache.org/jira/browse/METRON-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308152#comment-16308152 ] ASF GitHub Bot commented on METRON-1340: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/857 @justinleet I believe @iraghumitra is still working on the SELENIUM_PROMISE_MANAGER change proposed above but I will defer to him. > Improve e2e tests for metron alerts > --- > > Key: METRON-1340 > URL: https://issues.apache.org/jira/browse/METRON-1340 > Project: Metron > Issue Type: Bug >Reporter: RaghuMitra >Assignee: RaghuMitra > > Need to improve e2e tests in the following areas: > - Tests should not be flaky > - Remove the sleep ( This should implicitly make the tests run faster) > - Truncate HBase table 'metron_update' before starting the tests > - Improve the tests descriptions > - Run the tests headless if possible > - Check the node version and browser version before launching the tests > The expected behavior is that there are no intermittent failures. Acceptance > criteria: 5 consecutive runs without failures. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1381) Add Apache license to MD files and remove the Rat exclusion
[ https://issues.apache.org/jira/browse/METRON-1381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308139#comment-16308139 ] ASF GitHub Bot commented on METRON-1381: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/883 @nickwallen @ottobackwards I'm going to (possibly annoyingly) ping again and use the excuse that the holidays are over. > Add Apache license to MD files and remove the Rat exclusion > --- > > Key: METRON-1381 > URL: https://issues.apache.org/jira/browse/METRON-1381 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet > > We should be adding the Apache license in an HTML comment () on our > Markdown files. This allows the header to still be in the source, but hides > it from the rendering. I validated that this works on both the GitHub > rendering and the site-book rendering. > This came up in a discuss thread awhile ago, but it looks like it slipped > through the cracks afterwards. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1340) Improve e2e tests for metron alerts
[ https://issues.apache.org/jira/browse/METRON-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308127#comment-16308127 ] ASF GitHub Bot commented on METRON-1340: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/857 @merrimanr @mmiklavc @iraghumitra What are the next steps towards being able to have this stable? It sounds like there was some playing around with versions, and that there may be some more structural changes to the code style. How do you propose we move forward? I'd really like to see this stuff made stable and I'm happy to help out with more testing or whatever else we need. > Improve e2e tests for metron alerts > --- > > Key: METRON-1340 > URL: https://issues.apache.org/jira/browse/METRON-1340 > Project: Metron > Issue Type: Bug >Reporter: RaghuMitra >Assignee: RaghuMitra > > Need to improve e2e tests in the following areas: > - Tests should not be flaky > - Remove the sleep ( This should implicitly make the tests run faster) > - Truncate HBase table 'metron_update' before starting the tests > - Improve the tests descriptions > - Run the tests headless if possible > - Check the node version and browser version before launching the tests > The expected behavior is that there are no intermittent failures. Acceptance > criteria: 5 consecutive runs without failures. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1344) Externalize the infrastructural components using integration tests
[ https://issues.apache.org/jira/browse/METRON-1344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308111#comment-16308111 ] ASF GitHub Bot commented on METRON-1344: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/858 @cestella are you good with merging this in to the feature branch? I believe I addressed your comments. > Externalize the infrastructural components using integration tests > -- > > Key: METRON-1344 > URL: https://issues.apache.org/jira/browse/METRON-1344 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman > > As we move towards adding the Alerts UI e2e test to our continuous build, we > need to remove the dependence on vagrant full dev and provide equivalent > infrastructure within Travis. This will include all the services that the > e2e tests depend on including Kafka, Zookeeper, Elasticsearch (possibly > HBase), and Metron REST. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1233) Remove description of Global configuration from Enrichment doc
[ https://issues.apache.org/jira/browse/METRON-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308110#comment-16308110 ] ASF GitHub Bot commented on METRON-1233: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/789 @DimDroll Is this still relevant with Mike's comment about #791? > Remove description of Global configuration from Enrichment doc > -- > > Key: METRON-1233 > URL: https://issues.apache.org/jira/browse/METRON-1233 > Project: Metron > Issue Type: Improvement >Reporter: Dima Kovalyov >Priority: Minor > > It confuses reader. Description of Global configuration is available ahead of > the enrichment and listed here: > https://metron.apache.org/current-book/metron-platform/metron-common/index.html > Global config is used for validation of the fields. > Enrichment is used for fields enrichment and probably can also be used for > validation of the fields (which is not covered in the doc). > From available doc I can see that they serve two different purposes. > So, I don't see any reason to mention global config in enrichment > description. What's the benefit of knowing about global configuration when > you want deep dive in enrichment? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1231) Separate Sensor name and topic in the Management UI
[ https://issues.apache.org/jira/browse/METRON-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308104#comment-16308104 ] ASF GitHub Bot commented on METRON-1231: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/786 @merrimanr Can you deconflict this? This is otherwise ready for review, but it just needs to happen, right? > Separate Sensor name and topic in the Management UI > --- > > Key: METRON-1231 > URL: https://issues.apache.org/jira/browse/METRON-1231 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman > > Currently the Management UI (and REST app) assumes the sensor name == sensor > topic which is false. These should be handles separately and the UI should > expose a way to edit both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1005) Create Decodable Row Key for Profiler
[ https://issues.apache.org/jira/browse/METRON-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308099#comment-16308099 ] ASF GitHub Bot commented on METRON-1005: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/622 @nickwallen I haven't been following this discussion, but it seems like a useful feature / enhancement that's been hanging out awhile after active discussion petered out. What are the next steps here? Does this PR need changes? Should the discussion be revived on the user lists? It doesn't seem like there was any consensus on the approach, but again, I like this enhancement a lot. > Create Decodable Row Key for Profiler > - > > Key: METRON-1005 > URL: https://issues.apache.org/jira/browse/METRON-1005 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > To be able to answer the types of questions that I outlined in METRON-450, we > need a row key that is decodable. Right now there is no logic to decode a > row key, nor is the existing row key easily decodable. > Once the row keys can be decoded, you could scan all of the row keys in the > Profiler's HBase table, decode each of them and extract things like, the > names of all your profiles, the names of entities within a profile, the > period duration of a given profile. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1362) Improve Metron Deployment README
[ https://issues.apache.org/jira/browse/METRON-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308074#comment-16308074 ] ASF GitHub Bot commented on METRON-1362: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/869 @nickwallen Thanks for adding that. I'm good with it. > Improve Metron Deployment README > > > Key: METRON-1362 > URL: https://issues.apache.org/jira/browse/METRON-1362 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > > The README and docs under metron-deployment/ are rather confusing and > containing outdated information. I'd like to get that cleaned up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
