[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r15936 --- Diff: metron-deployment/packaging/ambari/metron-mpack/README.md --- @@ -0,0 +1,127 @@ + + +This provides a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, management and monitoring of Metron on clusters of any size. + +This allows you to easily install Metron using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + +### Prerequisites + +* Ambari 2.4.2+ + +* Installable Metron packages (either RPMs or DEBs) located in a repository on each host at `/localrepo`. + +* A [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the host running the Management and Alarm UI. + +### Quick Start + +1. Build the Metron MPack. Execute the following command from the project's root directory. +``` +mvn clean package -Pmpack -DskipTests +``` + +1. This results in the Mpack being produced at the following location. +``` + metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-x.y.z.0.tar.gz +``` + +1. Copy the tarball to the host where Ambari Server is installed. + +1. Ensure that Ambari Server is stopped. + +1. Install the MPack. +``` +ambari-server install-mpack --mpack=metron_mpack-x.y.z.0.tar.gz --verbose +``` + +1. Install the Metron packages (RPMs or DEBs) in a local repository on each host where a Metron component is installed. By default, the repository is expected to exist at `/localrepo`. + +On hosts where only a Metron client is installed, the local repository must exist, but it does not need to contain Metron packages. For example to create an empty repository for an RPM-based system, run the following commands. + +``` +yum install createrepo +mkdir /localrepo +cd /localrepo +createrepo +``` + +1. Metron will now be available as an installable service within Ambari. + +### Installation Notes + +The MPack will make all Metron services available in Ambari in the same manner as any other services in a stack. These can be installed using Ambari's user interface using "Add Services" or during an initial cluster install. + + Co-Location + +1. The Parsers, Enrichment, Indexing, and Profiler masters should be colocated on a host with a Kafka Broker. This is necessary so that the correct Kafka topics can be created. + +1. The Enrichment and Profiler masters should be colocated on a host with an HBase client. This is necessary so that the Enrichment, Threat Intel, and Profile tables can be created. + --- End diff -- Here are a couple more co-location requirements: 3. Enrichment, Indexing, Profiler should be co-located on the same host as Parsers. 4. Alerts UI and Management UI should be co-located on the same host. 5. Parsers and REST need to have an instance of STORM SUPERVISOR client ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159361598 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,134 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services +This project contains tools for building, packaging, and deploying Apache Metron. Please refer to the following sections for more information on how to get Apache Metron running in your environment. -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites -- A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](#rpms) for further information. -- [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the Management UI host - -### Building Management Pack -From `metron-deployment` run -``` -mvn clean package -``` - -A tar.gz that can be used with Ambari can be found at `metron-deployment/packaging/ambari/metron-mpack/target/` - -### Installing Management Pack -Before installing the mpack, update Storm's topology.classpath in Ambari to include '/etc/hbase/conf:/etc/hadoop/conf'. Restart Storm service. - -Place the mpack's tar.gz onto the node running Ambari Server. From the command line on this node, run -``` -ambari-server install-mpack --mpack= --verbose -``` - -This will make the services available in Ambari in the same manner as any services in a stack, e.g. through Add Services or during cluster install. -The Indexing / Parsers/ Enrichment masters should be colocated with a Kafka Broker (to create topics) and HBase client (to create the enrichment and theatintel tables). -This colocation is currently not enforced by Ambari, and should be managed by either a Service or Stack advisor as an enhancement. - -Several configuration parameters will need to be filled in, and should be pretty self explanatory (primarily a couple of Elasticsearch configs, and the Storm REST URL). Examples are provided in the descriptions on Ambari. -Notably, the URL for the GeoIP database that is preloaded (and is prefilled by default) can be set to use a `file:///` location - -After installation, a custom action is available in Ambari (where stop / start services are) to install Elasticsearc
[GitHub] metron issue #869: METRON-1362 Improve Metron Deployment README
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/869 +1, pending Travis. Good job, this should be really helpful! ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159330127 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,134 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services +This project contains tools for building, packaging, and deploying Apache Metron. Please refer to the following sections for more information on how to get Apache Metron running in your environment. -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites -- A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](#rpms) for further information. -- [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the Management UI host - -### Building Management Pack -From `metron-deployment` run -``` -mvn clean package -``` - -A tar.gz that can be used with Ambari can be found at `metron-deployment/packaging/ambari/metron-mpack/target/` - -### Installing Management Pack -Before installing the mpack, update Storm's topology.classpath in Ambari to include '/etc/hbase/conf:/etc/hadoop/conf'. Restart Storm service. - -Place the mpack's tar.gz onto the node running Ambari Server. From the command line on this node, run -``` -ambari-server install-mpack --mpack= --verbose -``` - -This will make the services available in Ambari in the same manner as any services in a stack, e.g. through Add Services or during cluster install. -The Indexing / Parsers/ Enrichment masters should be colocated with a Kafka Broker (to create topics) and HBase client (to create the enrichment and theatintel tables). -This colocation is currently not enforced by Ambari, and should be managed by either a Service or Stack advisor as an enhancement. - -Several configuration parameters will need to be filled in, and should be pretty self explanatory (primarily a couple of Elasticsearch configs, and the Storm REST URL). Examples are provided in the descriptions on Ambari. -Notably, the URL for the GeoIP database that is preloaded (and is prefilled by default) can be set to use a `file:///` location - -After installation, a custom action is available in Ambari (where stop / start services are) to install Elasticsearc
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159327372 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,134 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services +This project contains tools for building, packaging, and deploying Apache Metron. Please refer to the following sections for more information on how to get Apache Metron running in your environment. -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites -- A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](#rpms) for further information. -- [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the Management UI host - -### Building Management Pack -From `metron-deployment` run -``` -mvn clean package -``` - -A tar.gz that can be used with Ambari can be found at `metron-deployment/packaging/ambari/metron-mpack/target/` - -### Installing Management Pack -Before installing the mpack, update Storm's topology.classpath in Ambari to include '/etc/hbase/conf:/etc/hadoop/conf'. Restart Storm service. - -Place the mpack's tar.gz onto the node running Ambari Server. From the command line on this node, run -``` -ambari-server install-mpack --mpack= --verbose -``` - -This will make the services available in Ambari in the same manner as any services in a stack, e.g. through Add Services or during cluster install. -The Indexing / Parsers/ Enrichment masters should be colocated with a Kafka Broker (to create topics) and HBase client (to create the enrichment and theatintel tables). -This colocation is currently not enforced by Ambari, and should be managed by either a Service or Stack advisor as an enhancement. - -Several configuration parameters will need to be filled in, and should be pretty self explanatory (primarily a couple of Elasticsearch configs, and the Storm REST URL). Examples are provided in the descriptions on Ambari. -Notably, the URL for the GeoIP database that is preloaded (and is prefilled by default) can be set to use a `file:///` location - -After installation, a custom action is available in Ambari (where stop / start services are) to install Elasticsearc
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159327218 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,134 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services +This project contains tools for building, packaging, and deploying Apache Metron. Please refer to the following sections for more information on how to get Apache Metron running in your environment. -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites -- A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](#rpms) for further information. -- [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the Management UI host - -### Building Management Pack -From `metron-deployment` run -``` -mvn clean package -``` - -A tar.gz that can be used with Ambari can be found at `metron-deployment/packaging/ambari/metron-mpack/target/` - -### Installing Management Pack -Before installing the mpack, update Storm's topology.classpath in Ambari to include '/etc/hbase/conf:/etc/hadoop/conf'. Restart Storm service. - -Place the mpack's tar.gz onto the node running Ambari Server. From the command line on this node, run -``` -ambari-server install-mpack --mpack= --verbose -``` - -This will make the services available in Ambari in the same manner as any services in a stack, e.g. through Add Services or during cluster install. -The Indexing / Parsers/ Enrichment masters should be colocated with a Kafka Broker (to create topics) and HBase client (to create the enrichment and theatintel tables). -This colocation is currently not enforced by Ambari, and should be managed by either a Service or Stack advisor as an enhancement. - -Several configuration parameters will need to be filled in, and should be pretty self explanatory (primarily a couple of Elasticsearch configs, and the Storm REST URL). Examples are provided in the descriptions on Ambari. -Notably, the URL for the GeoIP database that is preloaded (and is prefilled by default) can be set to use a `file:///` location - -After installation, a custom action is available in Ambari (where stop / start services are) to install Elasticsearc
[GitHub] metron issue #831: METRON-1302: Split up Indexing Topology into batch and ra...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/831 I ran this up in full dev and everything worked as advertised. I only noticed a couple minor issues and left comments for those. I also am a little confused by the ra/batch vs es/hdfs issue. I still see places (Ambari config parameter names, Ambari MPack scripts, Flux file paths, Flux properties, Storm topology names, etc) where the prefixes are ra/batch and not es/hdfs. Reading through the PR comments I'm still not clear on what approach we decided on but I think consistency would be good. ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159324176 --- Diff: metron-deployment/packaging/ambari/metron-mpack/README.md --- @@ -0,0 +1,94 @@ + + +This provides a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, management and monitoring of Metron on clusters of any size. + +This allows you to easily install Metron using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + +### Prerequisites + +* Ambari 2.4.2+ + +* Installable Metron packages (either RPMs or DEBs) located in a repository on each host at `/localrepo`. + +* A [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the host running the Management and Alarm UI. + +### Quick Start + +1. Build the Metron MPack. Execute the following command from the project's root directory. +``` +mvn clean package -Pmpack -DskipTests +``` + +1. This results in the Mpack being produced at the following location. +``` + metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-x.y.z.0.tar.gz +``` + +1. Copy the tarball to the host where Ambari Server is installed. + +1. Ensure that Ambari Server is stopped. + +1. Install the MPack. +``` +ambari-server install-mpack --mpack=metron_mpack-x.y.z.0.tar.gz --verbose +``` + +1. Metron swill now be available as an installable service within Ambari. + +### Installation Notes + +The MPack will make all Metron services available in Ambari in the same manner as any other services in a stack. These can be installed using Ambari's user interface using "Add Services" or during an initial cluster install. + + Co-Location + +1. The Parsers, Enrichment, Indexing, and Profiler masters should be colocated on a host with a Kafka Broker. This is necessary so that the correct Kafka topics can be created. + +1. The Enrichment and Profiler masters should be colocated on a host with an HBase client. This is necessary so that the Enrichment, Threat Intel, and Profile tables can be created. + +This colocation is currently not enforced by Ambari and should be managed by either a Service or Stack advisor as an enhancement. + + Kerberization + +The MPack allows Metron to be automatically kerberized in two different ways. +* Metron can be installed on a non-kerberized cluster and then the entire cluster can be kerberized using Ambari. +* Alternatively, Metron can be installed on top of an already kerberized cluster. + +Using the MPack is preferred, but instructions for manually Kerberizing a cluster with Metron can be found at [Kerberos-manual-setup.md](../../Kerberos-manual-setup.md). + + Zeppelin Import + +A custom action is available in Ambari to import Zeppelin dashboards. See the [metron-indexing documentation](../metron-platform/metron-indexing) for more information. + + Offline Installation + +There is only one point during installation that reaches out to the internet. That is necessary to retrieve the GeoIP database. --- End diff -- I have had a hard time following the installation notes around the Mpack, so I definitely might have missed something there. I just went back and added some additional stuff. Hopefully that is everything that is still relevant. ---
[GitHub] metron pull request #831: METRON-1302: Split up Indexing Topology into batch...
Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/831#discussion_r159322924 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/hdfs.properties.j2 --- @@ -0,0 +1,44 @@ +# Licensed to the Apache Software Foundation (ASF) under one --- End diff -- Should this file be git ignored since it is copied in at build time? ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159322553 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,134 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services +This project contains tools for building, packaging, and deploying Apache Metron. Please refer to the following sections for more information on how to get Apache Metron running in your environment. -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites -- A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](#rpms) for further information. -- [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the Management UI host - -### Building Management Pack -From `metron-deployment` run -``` -mvn clean package -``` - -A tar.gz that can be used with Ambari can be found at `metron-deployment/packaging/ambari/metron-mpack/target/` - -### Installing Management Pack -Before installing the mpack, update Storm's topology.classpath in Ambari to include '/etc/hbase/conf:/etc/hadoop/conf'. Restart Storm service. - -Place the mpack's tar.gz onto the node running Ambari Server. From the command line on this node, run -``` -ambari-server install-mpack --mpack= --verbose -``` - -This will make the services available in Ambari in the same manner as any services in a stack, e.g. through Add Services or during cluster install. -The Indexing / Parsers/ Enrichment masters should be colocated with a Kafka Broker (to create topics) and HBase client (to create the enrichment and theatintel tables). -This colocation is currently not enforced by Ambari, and should be managed by either a Service or Stack advisor as an enhancement. - -Several configuration parameters will need to be filled in, and should be pretty self explanatory (primarily a couple of Elasticsearch configs, and the Storm REST URL). Examples are provided in the descriptions on Ambari. -Notably, the URL for the GeoIP database that is preloaded (and is prefilled by default) can be set to use a `file:///` location - -After installation, a custom action is available in Ambari (where stop / start services are) to install Elasticsearc
[GitHub] metron pull request #831: METRON-1302: Split up Indexing Topology into batch...
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/831#discussion_r159321881
--- Diff:
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/integration/IndexingIntegrationTest.java
---
@@ -197,9 +140,7 @@ public void test() throws Exception {
//assert that our input docs are equivalent to the output docs,
converting the input docs keys based
// on the field name converter
assertInputDocsMatchOutputs(inputDocs, docs,
getFieldNameConverter());
- assertInputDocsMatchOutputs(inputDocs, readDocsFromDisk(hdfsDir), x
-> x);
-} catch(Throwable e) {
- e.printStackTrace();
+ //assertInputDocsMatchOutputs(inputDocs, readDocsFromDisk(hdfsDir),
x -> x);
--- End diff --
Is this comment intentional?
---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159317149 --- Diff: metron-deployment/packaging/ambari/metron-mpack/README.md --- @@ -0,0 +1,94 @@ + + +This provides a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, management and monitoring of Metron on clusters of any size. + +This allows you to easily install Metron using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + +### Prerequisites + +* Ambari 2.4.2+ + +* Installable Metron packages (either RPMs or DEBs) located in a repository on each host at `/localrepo`. + +* A [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the host running the Management and Alarm UI. + +### Quick Start + +1. Build the Metron MPack. Execute the following command from the project's root directory. +``` +mvn clean package -Pmpack -DskipTests +``` + +1. This results in the Mpack being produced at the following location. +``` + metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-x.y.z.0.tar.gz +``` + +1. Copy the tarball to the host where Ambari Server is installed. + +1. Ensure that Ambari Server is stopped. + +1. Install the MPack. +``` +ambari-server install-mpack --mpack=metron_mpack-x.y.z.0.tar.gz --verbose +``` + +1. Metron swill now be available as an installable service within Ambari. + +### Installation Notes + +The MPack will make all Metron services available in Ambari in the same manner as any other services in a stack. These can be installed using Ambari's user interface using "Add Services" or during an initial cluster install. + + Co-Location + +1. The Parsers, Enrichment, Indexing, and Profiler masters should be colocated on a host with a Kafka Broker. This is necessary so that the correct Kafka topics can be created. + +1. The Enrichment and Profiler masters should be colocated on a host with an HBase client. This is necessary so that the Enrichment, Threat Intel, and Profile tables can be created. + +This colocation is currently not enforced by Ambari and should be managed by either a Service or Stack advisor as an enhancement. + + Kerberization + +The MPack allows Metron to be automatically kerberized in two different ways. +* Metron can be installed on a non-kerberized cluster and then the entire cluster can be kerberized using Ambari. +* Alternatively, Metron can be installed on top of an already kerberized cluster. + +Using the MPack is preferred, but instructions for manually Kerberizing a cluster with Metron can be found at [Kerberos-manual-setup.md](../../Kerberos-manual-setup.md). + + Zeppelin Import + +A custom action is available in Ambari to import Zeppelin dashboards. See the [metron-indexing documentation](../metron-platform/metron-indexing) for more information. + + Offline Installation + +There is only one point during installation that reaches out to the internet. That is necessary to retrieve the GeoIP database. --- End diff -- Sidenote, that should just be two slashes there (`file://`), since the extra `/` is the root dir, iirc. ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/869#discussion_r159316935
--- Diff: metron-deployment/packaging/ambari/metron-mpack/README.md ---
@@ -0,0 +1,94 @@
+
+
+This provides a Management Pack (MPack) extension for [Apache
Ambari](https://ambari.apache.org/) that simplifies the provisioning,
management and monitoring of Metron on clusters of any size.
+
+This allows you to easily install Metron using a simple, guided process.
This also allows you to monitor cluster health and even secure your cluster
with kerberos.
+
+### Prerequisites
+
+* Ambari 2.4.2+
+
+* Installable Metron packages (either RPMs or DEBs) located in a
repository on each host at `/localrepo`.
+
+* A [Node.js](https://nodejs.org/en/download/package-manager/) repository
installed on the host running the Management and Alarm UI.
+
+### Quick Start
+
+1. Build the Metron MPack. Execute the following command from the
project's root directory.
+```
+mvn clean package -Pmpack -DskipTests
+```
+
+1. This results in the Mpack being produced at the following location.
+```
+
metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-x.y.z.0.tar.gz
+```
+
+1. Copy the tarball to the host where Ambari Server is installed.
+
+1. Ensure that Ambari Server is stopped.
+
+1. Install the MPack.
+```
+ambari-server install-mpack --mpack=metron_mpack-x.y.z.0.tar.gz
--verbose
+```
+
+1. Metron swill now be available as an installable service within Ambari.
+
+### Installation Notes
+
+The MPack will make all Metron services available in Ambari in the same
manner as any other services in a stack. These can be installed using Ambari's
user interface using "Add Services" or during an initial cluster install.
+
+ Co-Location
+
+1. The Parsers, Enrichment, Indexing, and Profiler masters should be
colocated on a host with a Kafka Broker. This is necessary so that the correct
Kafka topics can be created.
+
+1. The Enrichment and Profiler masters should be colocated on a host with
an HBase client. This is necessary so that the Enrichment, Threat Intel, and
Profile tables can be created.
+
+This colocation is currently not enforced by Ambari and should be managed
by either a Service or Stack advisor as an enhancement.
+
+ Kerberization
+
+The MPack allows Metron to be automatically kerberized in two different
ways.
+* Metron can be installed on a non-kerberized cluster and then the entire
cluster can be kerberized using Ambari.
+* Alternatively, Metron can be installed on top of an already kerberized
cluster.
+
+Using the MPack is preferred, but instructions for manually Kerberizing a
cluster with Metron can be found at
[Kerberos-manual-setup.md](../../Kerberos-manual-setup.md).
+
+ Zeppelin Import
+
+A custom action is available in Ambari to import Zeppelin dashboards. See
the [metron-indexing documentation](../metron-platform/metron-indexing) for
more information.
+
+ Offline Installation
+
+There is only one point during installation that reaches out to the
internet. That is necessary to retrieve the GeoIP database.
--- End diff --
Some information definitely got lost here. There's a section that was
deleted ("Installing Management Pack") that mentioned:
> Notably, the URL for the GeoIP database that is preloaded (and is
prefilled by default) can be set to use a `file:///` location
Can you modify this part here, and take a quick swing through to see if
anything else got lost during the cleanup? The geo stuff showed up in a couple
places originally, which might be why it got chopped.
---
[GitHub] metron issue #622: METRON-1005 Create Decodable Row Key for Profiler
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/622 There was a lot of good discussion on this, but I find it hard to summarize completely the positions of everyone including @mattf-horton and @cestella. Here is my attempt in trying to do that. Please correct anything that I have misstated. 1. Everyone agreed that a ToC (table of contents) is a useful additional feature for the Profiler. The decodable row key would be needed in addition to, not instead of, a ToC. 1. In implementing a decodable row key, we do need to plan for future changes in row key format. This was handled in this PR, but can be improved. 1. The decodable row key feature should be completed **before** a ToC so that the row keys can be used to generate (or regenerate) a ToC on-demand. 1. There were various suggestions made on how to shorten up the row key format. Some of those I completed on this PR (like using a murmur hash) and others (like using shorts instead of ints) I would need to incorporate in a future PR for a decodable row key. 1. There is a need for a migration tool. A tool that can read the existing row key format and rewrite the same data using a new format. This tool is necessary even if it cannot be implemented deterministically with the current row key format. The tool may not hints from the user like the names of known profiles. Once I compile a summary of these changes, I will close this PR. All enhancements around this will be implemented on new PRs. ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159314084 --- Diff: metron-deployment/README.md --- @@ -1,175 +1,129 @@ -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + + How do I do it? + +# Prerequisites + - A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](
[GitHub] metron issue #869: METRON-1362 Improve Metron Deployment README
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/869 +1 Thanks ---
[GitHub] metron issue #786: METRON-1231: Separate Sensor name and topic in the Manage...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/786 @justinleet this should be ready for review now. ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159284348 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,132 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos.
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159284273 --- Diff: metron-deployment/README.md --- @@ -1,175 +1,129 @@ -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + + How do I do it? + +# Prerequisites + - A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](
Re: [DISCUSS] Removing Markdown files from rat exclusion
Bravo. On 1/2/18, 8:43 AM, "Justin Leet" wrote: The PR is merged into master, and all relevant PRs have a comment noting that adding the header is required. As a reminder, this means Apache headers are required on all markdown files and this will be enforced by rat. On Sat, Dec 30, 2017 at 8:33 AM, Justin Leet wrote: > I've updated the PR to add the header to a new MD file that went in. > > I've also commented on all PRs that I saw that would potentially be > problematic were they to go into master if they weren't merged first. > > Once the updated PR gets the +1's reaffirmed, it will be merged into > master and Markdown headers will be enforced properly going forwad. > > On Sun, Dec 24, 2017 at 8:09 PM, Justin Leet > wrote: > >> I'm gonna let this percolate until Wednesday or so, assuming conversation >> doesn't reach a natural tipping point. I'm inclined to agree with Nick, >> but I also don't want to resolve anything in a way that even potentially >> causes master problems until at least after Christmas has a chance to >> settle down for people. At that point, assuming current course, I'll take >> a real run through of the PRs (and leave comments as appropriate, before >> merging. >> >> Obviously if anyone has suggestions or alternatives, still feel >> encouraged to respond. >> >> On Sat, Dec 23, 2017 at 11:17 AM, Nick Allen wrote: >> >>> > This would result in master breaking (although it's a pretty easy fix). >>> >>> I am not concerned and don't think we need to wait on merging PR #883. >>> >>> Can you add a comment to each of the PRs that you identified? We can >>> make >>> sure that each gets merged with master before they go in. >>> >>> >>> >>> On Sat, Dec 23, 2017 at 11:08 AM, Justin Leet >>> wrote: >>> >>> > I have a PR currently out (https://github.com/apache/metron/pull/883) >>> that >>> > removes the rat exclusion on Markdown files. There was a discuss thread >>> > awhile back about adding the header and removing the exclusion where >>> it was >>> > agreed that we should do this to meet Apache requirements. >>> Unfortunately, >>> > it didn't get any follow on. >>> > >>> > Right now the PR has two +1s, but it could potentially be problematic >>> with >>> > existing PRs. >>> > >>> > Any PR that meets two conditions could potentially be problematic >>> > 1. It adds a new Markdown file >>> > 2. Travis was run before the exclusion PR was merged. >>> > >>> > This is because whoever does the merge might not realize that master >>> should >>> > be merged in and the markdown file updated with the Apache header. >>> This >>> > would result in master breaking (although it's a pretty easy fix). >>> > >>> > Are we okay with merging this now/soon, or do we want to take >>> additional >>> > steps to ensure we don't run into issues? If we want, I can run >>> through the >>> > PRs and add comments before merging. Is this sufficient to at least >>> > mitigate the most obvious problems? >>> > >>> > I took a very quick glance through some of the most recent PRs and >>> only two >>> > really stood out to me (although I'm sure there are older ones that are >>> > still being worked on or looked at) >>> > >>> > METRON-1380 https://github.com/apache/metron/pull/882 - Adds a new >>> > markdown >>> > file, but Travis failed. If it gets fixed before this PR is merged we >>> could >>> > run into the problem >>> > METRON-1351 https://github.com/apache/metron/pull/868 - Adds a new >>> > markdown >>> > file and Travis succeeded. This would break master if merged as-is >>> after my >>> > PR. >>> > >>> >> >> >
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user anandsubbu commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159280034 --- Diff: metron-deployment/README.md --- @@ -15,178 +15,132 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos.
[GitHub] metron pull request #885: METRON-1384 Increment master version number to 0.4...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/885 ---
[GitHub] metron issue #869: METRON-1362 Improve Metron Deployment README
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/869 Ok, @ottobackwards. I'll take a hack at that. ---
[GitHub] metron pull request #886: METRON-1385: Missing "properties" in index templat...
GitHub user merrimanr opened a pull request:
https://github.com/apache/metron/pull/886
METRON-1385: Missing "properties" in index template causes
ElasticsearchColumnMetadataDao.getColumnMetadata to fail
## Contributor Comments
A bug was recently discovered that causes a NPE when calling
ElasticsearchColumnMetadataDao.getColumnMetadata to get ES column types. It
can be recreated by adding a template to ES that matches all indices and also
contains a doc type mapping without the "properties" property. For example:
```
curl -XPUT 'http://node1:9200/_template/default_string_template' -d '
{
"template": "*",
"mappings" : {
"bro_type": {
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "text"
}
}
}
]
}
}
}
```
A NPE should now happen when calling the REST endpoint to get column
metadata for bro.
This PR fixes that bug by first checking to see if the "properties"
property exists and continuing on if it does not. I also added an additional
mapping in ElasticsearchSearchIntegrationTest that mimics this condition.
Happy to remove it if people feel it's unnecessary.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up
for your personal repository such that your branches are built there before
submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/merrimanr/incubator-metron METRON-1385
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/886.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #886
commit 8905d4d21926ac4e518d14fb0b4b44fbf4dea158
Author: merrimanr
Date: 2018-01-02T16:22:12Z
initial commit
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/840#discussion_r159273549
--- Diff: pom.xml ---
@@ -159,7 +159,7 @@
${global_surefire_version}
-@{argLine} -Xmx2048m
+-Xmx2048m
--- End diff --
This was in for code coverage via JaCoCo. It basically overrides the empty
argline from above that got deleted. Was it causing problems with running
things with it there?
---
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r159266301 --- Diff: metron-deployment/other-examples/manual-install/Manual_Install_CentOS6.md --- @@ -441,7 +441,7 @@ Client - Kibana: * Set "kibana_es_url" to `http://:9200`. "replace_with_elasticsearch_master_hostname" is the IP of the node where you assigned ElasticSearch Master on the Assign Master tab. -* Change kibana_default_application to "dashboard/Metron-Dashboard" +* Change kibana_default_application to "dashboard/AV-YpDmwdXwc6Ua9Muh9" --- End diff -- Is it possible to provide an easily identifiable name, or are we stuck with the "AV-..." ugliness? ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159268214 --- Diff: metron-deployment/README.md --- @@ -1,175 +1,129 @@ -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + + How do I do it? + +# Prerequisites + - A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](
[GitHub] metron issue #869: METRON-1362 Improve Metron Deployment README
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/869 https://github.com/apache/metron/tree/master/metron-deployment/packaging/docker/ansible-docker ---
[GitHub] metron issue #869: METRON-1362 Improve Metron Deployment README
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/869 The ansible-docker image can be used for this. ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159266396 --- Diff: metron-deployment/README.md --- @@ -1,175 +1,129 @@ -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + + How do I do it? + +# Prerequisites + - A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](
[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r159265938 --- Diff: metron-deployment/README.md --- @@ -1,3 +1,16 @@ +# Metron Deployment --- End diff -- As a heads up, #883 is in now, so this will have to be taken care of when you merge master to deconflict. ---
Re: [DISCUSS] Removing Markdown files from rat exclusion
The PR is merged into master, and all relevant PRs have a comment noting that adding the header is required. As a reminder, this means Apache headers are required on all markdown files and this will be enforced by rat. On Sat, Dec 30, 2017 at 8:33 AM, Justin Leet wrote: > I've updated the PR to add the header to a new MD file that went in. > > I've also commented on all PRs that I saw that would potentially be > problematic were they to go into master if they weren't merged first. > > Once the updated PR gets the +1's reaffirmed, it will be merged into > master and Markdown headers will be enforced properly going forwad. > > On Sun, Dec 24, 2017 at 8:09 PM, Justin Leet > wrote: > >> I'm gonna let this percolate until Wednesday or so, assuming conversation >> doesn't reach a natural tipping point. I'm inclined to agree with Nick, >> but I also don't want to resolve anything in a way that even potentially >> causes master problems until at least after Christmas has a chance to >> settle down for people. At that point, assuming current course, I'll take >> a real run through of the PRs (and leave comments as appropriate, before >> merging. >> >> Obviously if anyone has suggestions or alternatives, still feel >> encouraged to respond. >> >> On Sat, Dec 23, 2017 at 11:17 AM, Nick Allen wrote: >> >>> > This would result in master breaking (although it's a pretty easy fix). >>> >>> I am not concerned and don't think we need to wait on merging PR #883. >>> >>> Can you add a comment to each of the PRs that you identified? We can >>> make >>> sure that each gets merged with master before they go in. >>> >>> >>> >>> On Sat, Dec 23, 2017 at 11:08 AM, Justin Leet >>> wrote: >>> >>> > I have a PR currently out (https://github.com/apache/metron/pull/883) >>> that >>> > removes the rat exclusion on Markdown files. There was a discuss thread >>> > awhile back about adding the header and removing the exclusion where >>> it was >>> > agreed that we should do this to meet Apache requirements. >>> Unfortunately, >>> > it didn't get any follow on. >>> > >>> > Right now the PR has two +1s, but it could potentially be problematic >>> with >>> > existing PRs. >>> > >>> > Any PR that meets two conditions could potentially be problematic >>> > 1. It adds a new Markdown file >>> > 2. Travis was run before the exclusion PR was merged. >>> > >>> > This is because whoever does the merge might not realize that master >>> should >>> > be merged in and the markdown file updated with the Apache header. >>> This >>> > would result in master breaking (although it's a pretty easy fix). >>> > >>> > Are we okay with merging this now/soon, or do we want to take >>> additional >>> > steps to ensure we don't run into issues? If we want, I can run >>> through the >>> > PRs and add comments before merging. Is this sufficient to at least >>> > mitigate the most obvious problems? >>> > >>> > I took a very quick glance through some of the most recent PRs and >>> only two >>> > really stood out to me (although I'm sure there are older ones that are >>> > still being worked on or looked at) >>> > >>> > METRON-1380 https://github.com/apache/metron/pull/882 - Adds a new >>> > markdown >>> > file, but Travis failed. If it gets fixed before this PR is merged we >>> could >>> > run into the problem >>> > METRON-1351 https://github.com/apache/metron/pull/868 - Adds a new >>> > markdown >>> > file and Travis succeeded. This would break master if merged as-is >>> after my >>> > PR. >>> > >>> >> >> >
[GitHub] metron pull request #883: METRON-1381: Add Apache license to MD files and re...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/883 ---
[GitHub] metron issue #883: METRON-1381: Add Apache license to MD files and remove th...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/883 +1 ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159261149 --- Diff: metron-deployment/README.md --- @@ -1,175 +1,129 @@ -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + + How do I do it? + +# Prerequisites + - A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](
[GitHub] metron issue #869: METRON-1362 Improve Metron Deployment README
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/869 Do those instructions exist somewhere before this PR ? That sounds like new functionality or am I misunderstanding your request? ---
[GitHub] metron issue #885: METRON-1384 Increment master version number to 0.4.3 for ...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/885 I'll take care of getting this merged now. ---
[GitHub] metron issue #883: METRON-1381: Add Apache license to MD files and remove th...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/883 I'm +1 ---
[GitHub] metron issue #869: METRON-1362 Improve Metron Deployment README
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/869 Can we have a section on `How dow I build everything with docker` with instructions for the ansible docker container? ---
[GitHub] metron issue #885: METRON-1384 Increment master version number to 0.4.3 for ...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/885 I'm +1 by inspection, since Nick built the code base. Thanks for taking care of this! ---
[GitHub] metron issue #885: METRON-1384 Increment master version number to 0.4.3 for ...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/885 +1 Thanks, Matt! Manually reviewed the changes and built the code base. All remaining references to 0.4.2 are part of step-by-step instructions where the version does not really need changed. ---
[GitHub] metron pull request #869: METRON-1362 Improve Metron Deployment README
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/869#discussion_r159246961 --- Diff: metron-deployment/README.md --- @@ -1,175 +1,129 @@ -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. - -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services - -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites +This project contains a variety of tools for deploying Apache Metron. Please refer to the following sections for more information on the best way to deploy Metron in your environment. + + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I deploy Metron on a large cluster with Ambari?](#how-do-i-deploy-metron-on-a-large-cluster-with-ambari) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + +## How do I deploy Metron on a single VM? + + Who is this for? + +If you are new to Metron and just want to explore the functionality that it offers, this is good place to start. If you are a developer building on Metron, then this is also a great way to test out your changes. + + What does this do? + +This option will deploy Metron on a virtual machine running on your computer. This is often referred to as the "Full Dev" environment. + + How do I do it? +Follow these [instructions to deploy Metron on a single VM](vagrant/full-dev-platform/README.md). + + Where can I get more information? +* [Full Dev Platform](vagrant/full-dev-platform) + +## How do I deploy Metron on a large cluster with Ambari? + + Who is this for? + +If you want to see how Metron can really scale by deploying it on your own hardware, or even in your own cloud-based environment, this is the best option for you. + + What does this do? + +This creates a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, managing and monitoring of Metron on large clusters. This allows you to easily install Metron on a large cluster using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + + How do I do it? + +# Prerequisites + - A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](
[GitHub] metron issue #869: METRON-1362 Improve Metron Deployment README
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/869 @justinleet Is that a +1 then? @ottobackwards What are your thoughts on this PR? Is this an incremental improvement that we can merge? ---
[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/857 @justinleet I believe @iraghumitra is still working on the SELENIUM_PROMISE_MANAGER change proposed above but I will defer to him. ---
[GitHub] metron issue #883: METRON-1381: Add Apache license to MD files and remove th...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/883 @nickwallen @ottobackwards I'm going to (possibly annoyingly) ping again and use the excuse that the holidays are over. ---
[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/857 @merrimanr @mmiklavc @iraghumitra What are the next steps towards being able to have this stable? It sounds like there was some playing around with versions, and that there may be some more structural changes to the code style. How do you propose we move forward? I'd really like to see this stuff made stable and I'm happy to help out with more testing or whatever else we need. ---
[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/858 @cestella are you good with merging this in to the feature branch? I believe I addressed your comments. ---
[GitHub] metron issue #789: METRON-1233: Remove description of Global configuration f...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/789 @DimDroll Is this still relevant with Mike's comment about #791? ---
[GitHub] metron issue #786: METRON-1231: Separate Sensor name and topic in the Manage...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/786 @merrimanr Can you deconflict this? This is otherwise ready for review, but it just needs to happen, right? ---
[GitHub] metron issue #622: METRON-1005 Create Decodable Row Key for Profiler
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/622 @nickwallen I haven't been following this discussion, but it seems like a useful feature / enhancement that's been hanging out awhile after active discussion petered out. What are the next steps here? Does this PR need changes? Should the discussion be revived on the user lists? It doesn't seem like there was any consensus on the approach, but again, I like this enhancement a lot. ---
[GitHub] metron issue #869: METRON-1362 Improve Metron Deployment README
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/869 @nickwallen Thanks for adding that. I'm good with it. ---
Re: [DISCUSS] Resources for how to contribute to Apache Metron
Bump Anyone have any thoughts? On December 20, 2017 at 10:37:03, Casey Stella (ceste...@gmail.com) wrote: That's really good feedback, Jon. I agree that we have a significant barrier to get to the point of tinkering. Full-dev really wasn't intended to be that entry point; it's more of a way to test PRs in something resembling a realistic scenario (and it is still not super realistic). I would welcome creative ideas around how to accomplish that goal. On Wed, Dec 20, 2017 at 10:15 AM, zeo...@gmail.com wrote: > For nearly everybody I've talked to about this project that had complaints, > I've heard something about the significant barrier to entry, divided into > two general categories. Category 1 is that a lot of security teams lack > substantial experience with Hadoop and would like to get a better > understanding of how the involved components fit together - not > just kafka goes to storm goes to kafka, or a link to the kafka docs for > details about kafka, but a little bit more detail as to _why_ those > components are in use in metron, what properties those components possess > at a high level _which makes them appealing to us_, and how they're > _currently used_ in the metron environment. Category 2 is that it is > generally more difficult than it should be to get a testing/poc environment > running - running it on a laptop (especially non-macOS) can be a pain to > get running, some laptops simply cannot run it, etc. I've heard a few > times that a company uses Azure (not AWS) and they would like to quickly > spin it up there. > > Just my $0.02 > > Jon > > On Tue, Dec 19, 2017 at 9:02 AM Otto Fowler > wrote: > > > Like any project, Apache Metron needs to maintain and grow it’s > contributor > > community. We think that we could be doing a better job of this, and > would > > like to discuss issues and possible improvements. Issues > > > > What are some of the issues that may inhibit people contributing? > > > > - Barrier of entry (issues getting Metron running in vagrant or local) > > - Documentation : finding current > > - Documentation : content and quality > > - Source Code navigation/documentation/guides > > - Testing guides > > - Use Case Guides > > - Don’t know how they *can* contribute > > - Others that I’m missing? > > > > Remediation Barrier of entry > > > > How can we make the local deployment workflow easier ( other discuss > thread > > touches on this)? > > Documentation : Finding Current > > > > When people look for Metron info, where are they looking? What comes up > in > > search? - Hortonworks Community forums ( preview release stuff ? ), old > > blog posts? - Mailing list archives? - wiki? (not current) - site-book? > > > > How can we reduce the out of data information, and make the relevant > > information more prominent? > > Documentation : Content and Quality > > > > ( this is a little bit of a chicken and egg issue, since documentation > is a > > wonderful way to contribute…. ) - Up to data architecture documentation - > > Non-developer focused ‘feature’ documentation - Developer focused > > documentation ( how to add a XX guides ) > > Source Code Guides > > > > - Structure of the code tree > > - What is where, how it is logically setup > > - How to maintain concistancy when working in the code > > - Javadoc > > > > Testing Guides > > > > - Tests that we have are buried in PR’s > > - No regression tests > > > > Use case guides > > > > - more how-to guides > > > > Contributing guide > > > > - right now, have dev env guide > > - review and submit doc changes > > - review PR guide > > - pr testing guide ( better pr testing steps?) > > > > These are things I can think of, anyone have any comment, additions, > > priorities? > > > -- > > Jon >
