metron git commit: METRON-1801 Allow Customization of Elasticsearch Document ID (nickwallen) closes apache/metron#1218
Repository: metron
Updated Branches:
refs/heads/master 9b6710053 -> 90c5e1d28
METRON-1801 Allow Customization of Elasticsearch Document ID (nickwallen)
closes apache/metron#1218
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/90c5e1d2
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/90c5e1d2
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/90c5e1d2
Branch: refs/heads/master
Commit: 90c5e1d2896b8a95bd6c928e3dab96f1fd85167c
Parents: 9b67100
Author: nickwallen
Authored: Thu Oct 11 16:29:40 2018 -0400
Committer: nickallen
Committed: Thu Oct 11 16:29:40 2018 -0400
--
Upgrading.md| 23 ++-
.../METRON/CURRENT/configuration/metron-env.xml | 9 ++
.../CURRENT/package/scripts/metron_service.py | 5 +
.../package/scripts/params/params_linux.py | 1 +
.../METRON/CURRENT/themes/metron_theme.json | 10 ++
metron-platform/metron-elasticsearch/README.md | 37 -
metron-platform/metron-elasticsearch/pom.xml| 17 ---
.../dao/ElasticsearchRetrieveLatestDao.java | 147 ---
.../dao/ElasticsearchSearchDao.java | 28 +++-
.../elasticsearch/utils/ElasticsearchUtils.java | 100 +
.../writer/ElasticsearchWriter.java | 39 +++--
.../elasticsearch/dao/ElasticsearchDaoTest.java | 10 +-
.../ElasticsearchSearchIntegrationTest.java | 20 ++-
.../indexing/dao/SearchIntegrationTest.java | 21 +--
.../integration/SolrSearchIntegrationTest.java | 19 +++
15 files changed, 341 insertions(+), 145 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/90c5e1d2/Upgrading.md
--
diff --git a/Upgrading.md b/Upgrading.md
index 2124ac5..c3b2a0f 100644
--- a/Upgrading.md
+++ b/Upgrading.md
@@ -16,9 +16,22 @@ See the License for the specific language governing
permissions and
limitations under the License.
-->
# Upgrading
+
This document constitutes a per-version listing of changes of
configuration which are non-backwards compatible.
+## 0.6.0 to 0.6.1
+
+### [METRON-1801 Allow Customization of Elasticsearch Document
ID](https://issues.apache.org/jira/browse/METRON-1801)
+
+A global property named `es.document.id` was added to define the field from
which the document ID is set when a message is indexed by Elasticsearch. To
allow Elasticsearch to define its own document id, this property should be set
to a blank or empty string. The client will not set the document ID and
Elasticsearch will define its own. In most cases allowing Elasticsearch to
define the document ID is the most performant option. This is now the default
behavior.
+
+Metron versions 0.6.0 and earlier defined the document ID using the Metron
GUID, which is a randomized UUID using Java's `UUID.randomUUID()`. Using a
randomized UUID can negatively impact Elasticsearch indexing performance. To
maintain backwards compatibility with legacy versions of Metron use the
following global property setting.
+
+```
+es.document.id = guid
+```
+
## 0.4.2 to 0.5.0
### [METRON-941: native PaloAlto parser corrupts message when having a comma
in the payload](https://issues.apache.org/jira/browse/METRON-941)
@@ -89,7 +102,7 @@ For a more detailed description, please see
metron-platform/metron-elasticsearch
### Description
-In the 0.4.2 release,
+In the 0.4.2 release,
## 0.3.1 to 0.4.0
@@ -107,7 +120,7 @@ This effectively limits the build environment to Docker
supported [platforms](ht
Description
As of 0.3.0 the indexing configuration
-* Is held in the enrichment configuration for a sensor
+* Is held in the enrichment configuration for a sensor
* Has properties which control every writers (i.e. HDFS, solr or
elasticsearch).
In the 0.3.1 release, this configuration has been broken out
@@ -136,7 +149,7 @@ You would create a file to configure each writer for sensor
`foo` called `$METRO
"batchSize" : 100,
"enabled" : true
},
- "hdfs" : {
+ "hdfs" : {
"index" : "foo",
"batchSize" : 100,
"enabled" : true
@@ -151,7 +164,7 @@ You would create a file to configure each writer for sensor
`foo` called `$METRO
As of 0.3.0, threat triage rules were defined as a simple Map associating a
Stellar expression with a score.
As of 0.3.1, due to the fact that there may be many threat triage rules, we
have made the rules more complex.
To help organize these, we have made the threat triage objects in their own
right that contain optional name and optional comment fields.
-
+
This essentially makes the risk level rules slightly more complex. The format
goes from:
```
"riskLevelRules" : {
@@ -169,7 +182,7 @@ to:
}
]
```
-
+
Migration
For every sensor
metron git commit: METRON-1799: Remove outdated bylaws from site.
Repository: metron
Updated Branches:
refs/heads/asf-site 58d422b9c -> 572e6decf
METRON-1799: Remove outdated bylaws from site.
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/572e6dec
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/572e6dec
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/572e6dec
Branch: refs/heads/asf-site
Commit: 572e6decf4f1787d2fbfb212dee839c864038b8f
Parents: 58d422b
Author: justinjleet
Authored: Thu Oct 11 08:52:29 2018 -0400
Committer: justinjleet
Committed: Thu Oct 11 08:52:29 2018 -0400
--
develop/bylaws.html | 509 -
develop/bylaws/index.html | 509 -
develop/coding.html | 244
develop/coding/index.html | 244
develop/index.html| 324 --
feed.xml | 6 +-
6 files changed, 3 insertions(+), 1833 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/572e6dec/develop/bylaws.html
--
diff --git a/develop/bylaws.html b/develop/bylaws.html
deleted file mode 100644
index 16fdbd9..000
--- a/develop/bylaws.html
+++ /dev/null
@@ -1,509 +0,0 @@
-
-
-http://ogp.me/ns#;
ng-app="hortonWorksApp">@charset
"UTF-8";[ng\:cloak],[ng-cloak],[data-ng-cloak],[x-ng-cloak],.ng-cloak,.x-ng-cloak,.ng-hide:not(.ng-hide-animate){display:none
!important;}ng\:form{display:block;}.ng-animate-shim{visibility:hidden;}.ng-anchor{position:absolute;}
-
-
-http://gmpg.org/xfn/11;>
-Apache Metron Bylaws
-
-
-img.wp-smiley,
-img.emoji {
-display: inline !important;
-border: none !important;
-box-shadow: none !important;
-height: 1em !important;
-width: 1em !important;
-margin: 0 .07em !important;
-vertical-align: -0.1em !important;
-background: none !important;
-padding: 0 !important;
-}
-
-
-
-
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
-(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new
Date();a=s.createElement(o),
-
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
-
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
-
-ga('create', 'UA-78972211-1', 'auto');
-ga('send', 'pageview');
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-window.lsfDataLayer = window.lsfDataLayer || [];
-
-
-
- body {
-font-family: "roboto-condensed", sans-serif;
- }
-
-
-
-
-
-
-
-
-
-
-
-
-
-/* Copyright 2014 Evernote Corporation. All rights reserved. */
-
-.en-markup-crop-options {
-top: 18px !important;
-left: 50% !important;
-margin-left: -100px !important;
-width: 200px !important;
-border: 2px rgba(255, 255, 255, .38) solid !important;
-border-radius: 4px !important;
-}
-
-.en-markup-crop-options div div:first-of-type {
-margin-left: 0px !important;
-}
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Home
-
-About
-
-Documentation
-
-Community
-
-News
-
-
-
Download
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-Apache Metron Bylaws
-Introduction
-
-This document defines the bylaws under which the Apache Metron project
-operates. It defines the roles and responsibilities of the project,
-who may vote, how voting works, how conflicts are resolved, etc.
-
-Metron is a project of the Apache Software Foundation (ASF) and the
foundation
-holds the trademark on the name âMetronâ and copyright on the combined
-code base. The http://www.apache.org/foundation/faq.html;>Apache
Foundation
-FAQ and
-http://www.apache.org/foundation/how-it-works.html;>How-It-Works
-explain the operation and background of the foundation.
-
-Apache has a http://www.apache.org/foundation/policies/conduct.html;>code of
metron git commit: METRON-1799 Remove outdated bylaws from site. (justinleet) closes apache/metron#1216
Repository: metron Updated Branches: refs/heads/master f153375fa -> 9b6710053 METRON-1799 Remove outdated bylaws from site. (justinleet) closes apache/metron#1216 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/9b671005 Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/9b671005 Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/9b671005 Branch: refs/heads/master Commit: 9b6710053894d8a39880cb8157a0e603ed542cb7 Parents: f153375 Author: justinleet Authored: Thu Oct 11 08:41:23 2018 -0400 Committer: leet Committed: Thu Oct 11 08:41:23 2018 -0400 -- site/develop/bylaws.md | 281 site/develop/coding.md | 34 -- site/develop/index.md | 61 -- 3 files changed, 376 deletions(-) -- http://git-wip-us.apache.org/repos/asf/metron/blob/9b671005/site/develop/bylaws.md -- diff --git a/site/develop/bylaws.md b/site/develop/bylaws.md deleted file mode 100644 index a8fc5fb..000 --- a/site/develop/bylaws.md +++ /dev/null @@ -1,281 +0,0 @@ -layout: page -title: Apache Metron Bylaws - -## Introduction - -This document defines the bylaws under which the Apache Metron project -operates. It defines the roles and responsibilities of the project, -who may vote, how voting works, how conflicts are resolved, etc. - -Metron is a project of the Apache Software Foundation (ASF) and the foundation -holds the trademark on the name "Metron" and copyright on the combined -code base. The [Apache Foundation -FAQ](http://www.apache.org/foundation/faq.html) and -[How-It-Works](http://www.apache.org/foundation/how-it-works.html) -explain the operation and background of the foundation. - -Apache has a [code of -conduct](http://www.apache.org/foundation/policies/conduct.html) that -it expects its members to follow. In particular: - -* Be **open and welcoming**. It is important that we grow and - encourage the community of users and developers for our project. - -* Be **collaborative**. Working together on the open mailing lists and - bug database to make decisions helps the project grow. - -* Be **respectful** of others. Everyone is volunteering their time and - efforts to work on this project. Please be respectful of everyone - and their views. - -Metron is typical of Apache projects in that it operates under a set of -principles, known collectively as the "Apache Way". If you are new to -Apache development, please refer to -[this](http://www.apache.org/foundation/how-it-works.html) for more -information on how Apache projects operate. - -## Roles and Responsibilities - -Apache projects define a set of roles with associated rights and -responsibilities. These roles govern what tasks an individual may -perform within the project. The roles are defined in the following -sections. - -### Users - -The most important participants in the project are people who use our -software. The majority of our developers start out as users and guide -their development efforts from the user's perspective. Users -contribute to the Apache projects by providing feedback to developers -in the form of bug reports and feature suggestions. As well, users -participate in the Apache community by helping other users on mailing -lists and user support forums. - -### Contributors - -Contributors include all of the volunteers who donate time, code, -documentation, -or resources to the Metron Project. A contributor that makes sustained, -welcome contributions to the project may be invited to become a -committer, though the exact timing of such invitations depends on many -factors. - -### Committers - -The project's committers are responsible for the project's technical -management. Committers have the right to commit to the project's git -repository. Committers may cast binding votes on any technical -discussion. - -Committer access is by invitation only and must be approved by -consensus approval of the active Project Management Committee (PMC) -members. - -If a committer wishes to leave the project or does not contribute to -the project in any form for six months, the PMC may make them emeritus. -Emeritus committers lose their ability to commit code or cast binding -votes. An emeritus committer may -request reinstatement of commit access from the PMC. Such -reinstatement is subject to consensus approval of active PMC members. - -All Apache committers are required to have a signed [Individual -Contributor License -Agreement](https://www.apache.org/licenses/icla.txt) (ICLA) on file -with the Apache Software Foundation. There is a [Committer -FAQ](http://www.apache.org/dev/committers.html) which provides more -details on the requirements for Committers. - -A committer who makes a -sustained contribution
metron-bro-plugin-kafka git commit: METRON-1800 Increment metron-bro-plugin-kafka version (JonZeolla via jonzeolla) closes apache/metron-bro-plugin-kafka#10
Repository: metron-bro-plugin-kafka Updated Branches: refs/heads/master 38f45e2bb -> d1a09b6a5 METRON-1800 Increment metron-bro-plugin-kafka version (JonZeolla via jonzeolla) closes apache/metron-bro-plugin-kafka#10 Project: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/repo Commit: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/commit/d1a09b6a Tree: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/tree/d1a09b6a Diff: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/diff/d1a09b6a Branch: refs/heads/master Commit: d1a09b6a50f20e5fa6cf5c758eea8b0d39ce65be Parents: 38f45e2 Author: JonZeolla Authored: Thu Oct 11 07:20:32 2018 -0400 Committer: jonzeolla Committed: Thu Oct 11 07:20:32 2018 -0400 -- README.md | 2 +- VERSION | 2 +- bro-pkg.meta| 2 +- src/Plugin.cc | 2 +- tests/Baseline/kafka.show-plugin/output | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/d1a09b6a/README.md -- diff --git a/README.md b/README.md index bc2faf4..91b6e1b 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ This software is a part of the [Apache Metron](http://metron.apache.org/) projec ``` $ bro -N Apache::Kafka -Apache::Kafka - Writes logs to Kafka (dynamic, version 0.2) +Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3) ``` ## Activation http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/d1a09b6a/VERSION -- diff --git a/VERSION b/VERSION index e59aa6f..c30be1e 100644 --- a/VERSION +++ b/VERSION @@ -15,4 +15,4 @@ # limitations under the License. # -0.2 +0.3 http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/d1a09b6a/bro-pkg.meta -- diff --git a/bro-pkg.meta b/bro-pkg.meta index 3d0cfa0..095ed5b 100644 --- a/bro-pkg.meta +++ b/bro-pkg.meta @@ -6,7 +6,7 @@ build_command = ./configure --bro-dist=%(bro_dist)s --with-librdkafka=%(LIBRDKAF test_command = ( cd tests && btest -d ) plugin_dir = build config_files = scripts/init.bro -version = 0.2 +version = 0.3 depends = bro >=2.5.0 bro-pkg >=1.2 http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/d1a09b6a/src/Plugin.cc -- diff --git a/src/Plugin.cc b/src/Plugin.cc index ed0cb39..a981d22 100644 --- a/src/Plugin.cc +++ b/src/Plugin.cc @@ -31,6 +31,6 @@ plugin::Configuration Plugin::Configure() config.name = "Apache::Kafka"; config.description = "Writes logs to Kafka"; config.version.major = 0; -config.version.minor = 2; +config.version.minor = 3; return config; } http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/d1a09b6a/tests/Baseline/kafka.show-plugin/output -- diff --git a/tests/Baseline/kafka.show-plugin/output b/tests/Baseline/kafka.show-plugin/output index 1094ebb..814421c 100644 --- a/tests/Baseline/kafka.show-plugin/output +++ b/tests/Baseline/kafka.show-plugin/output @@ -1,4 +1,4 @@ -Apache::Kafka - Writes logs to Kafka (dynamic, version 0.2) +Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3) [Writer] KafkaWriter (Log::WRITER_KAFKAWRITER) [Constant] Kafka::kafka_conf [Constant] Kafka::topic_name
