[metron-bro-plugin-kafka] Git Push Summary
Repository: metron-bro-plugin-kafka Updated Tags: refs/tags/0.2 [created] 1dfc5239f
metron git commit: METRON-1769 Script creation of a release candidate (justinleet) closes apache/metron#1188
Repository: metron Updated Branches: refs/heads/master 3d923cde8 -> f153375fa METRON-1769 Script creation of a release candidate (justinleet) closes apache/metron#1188 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/f153375f Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/f153375f Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/f153375f Branch: refs/heads/master Commit: f153375fad65e75a10ce490b4a5f29f7e82f70dd Parents: 3d923cd Author: justinleet Authored: Wed Oct 10 15:41:22 2018 -0400 Committer: leet Committed: Wed Oct 10 15:41:22 2018 -0400 -- dev-utilities/release-utils/README.md | 129 +++ dev-utilities/release-utils/metron-rc-check | 6 +- .../release-utils/prepare-release-candidate | 337 +++ 3 files changed, 469 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/metron/blob/f153375f/dev-utilities/release-utils/README.md -- diff --git a/dev-utilities/release-utils/README.md b/dev-utilities/release-utils/README.md new file mode 100644 index 000..d1fd816 --- /dev/null +++ b/dev-utilities/release-utils/README.md @@ -0,0 +1,129 @@ + + +# Release Tools + +This project contains tools to assist Apache Metron project committers. + +## Prepare Release Candidate + +This script automates the process of creating a release candidate from `apache/metron` or `apache/metron-bro-plugin-kafka`. The script will prompt for various information necessary. Ensure your signing key is setup per [Release Signing](https://www.apache.org/dev/release-signing.html) and [Apache GnuPGP Instructions](https://www.apache.org/dev/openpgp.html#gnupg) + +When prompted the `[value in brackets]` is used by default. To accept the default, simply press `enter`. If you would like to change the default, type it in and hit `enter` when done. + +In the following example, enter the appropriate information + +1. Execute the script. + +The first time the script is run, you will be prompted for additional information including your Apache username and Apache email. These values are persisted in `~/.metron-prepare-release-candidate`. Subsequent executions of the script will retrieve these values, rather than prompting you again for them. + +``` +$ ./prepare-release-candidate + your apache userid []: leet + your apache email [l...@apache.org]: +``` + +1. Select a repository we're creating an RC for. + +``` +[1] metron +[2] metron-bro-plugin-kafka + which repo? [1]: 1 +``` + +1. Enter the current version number. This will be the base for the CHANGES file + +``` + current version: 0.6.0 +``` + +1. Enter the version being built. + +``` + version being built: 0.6.1 +``` + +1. Enter the current RC number + +``` + release candidate number: 1 +``` + +1. Enter the branch we're releasing from. In most cases, this will be master, but for maintenance releases it can be another branch. + +``` + base revision branch or hash for release candidate [master]: +``` + +1. Enter the signing key id. + +``` + signing key id in 8-byte format (e.g. BADDCAFEDEADBEEF): +``` + +1. Enter if this is a practice run. In a practice run, nothing is pushed to SVN, but everything is setup and built otherwise. + +``` + do a live run (push to remote repositories?) [y/n] +``` + +1. Wait for all repos to be checked out to complete. There will be some additional work done, e.g. along with branch and tag creation. In a live run, you may be prompted for Git credentials to push a branch. + +``` + Checking out repo: https://dist.apache.org/repos/dist/dev/metron + Checking out repo: dev + Checking out repo: https://dist.apache.org/repos/dist/release/metron + Checking out repo: release + Checking out git repo: https://git-wip-us.apache.org/repos/asf/metron.git + Cloning into '/Users/justinleet/tmp/metron-0.6.1/metron'... + remote: Counting objects: 46146, done. + remote: Compressing objects: 100% (15568/15568), done. + remote: Total 46146 (delta 21513), reused 43696 (delta 19489) + Receiving objects: 100% (46146/46146), 56.00 MiB | 1.04 MiB/s, done. + Resolving deltas: 100% (21513/21513), done. + Creating branch: Metron_0.6.1 + Using git rev: master + Already on 'master' + Your branch is up to date with 'origin/master'. + Switched to a new branch 'Metron_0.6.1' + This is a practice run. Not running + Creating tentative git tag <0.6.1-rc1>. Do not push this tag until RC is ready for community review. + Already on 'Metron_0.6.1' + Creating the RC tarball for tag
metron git commit: METRON-1761 Allow a grok statement to be applied to each line in a file. (ottobackwards) closes apache/metron#1184
Repository: metron
Updated Branches:
refs/heads/master 060d17e99 -> 3d923cde8
METRON-1761 Allow a grok statement to be applied to each line in a file.
(ottobackwards) closes apache/metron#1184
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/3d923cde
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/3d923cde
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/3d923cde
Branch: refs/heads/master
Commit: 3d923cde8711dcc463c2d27cb2b67275b9172112
Parents: 060d17e
Author: ottobackwards
Authored: Wed Oct 10 11:31:31 2018 -0400
Committer: otto
Committed: Wed Oct 10 11:31:31 2018 -0400
--
.../src/main/sample/patterns/test | 1 +
metron-platform/metron-parsers/README.md| 10 +-
.../message-parser-implementation-notes.md | 57 +
.../parsers/DefaultMessageParserResult.java | 76 +++
.../org/apache/metron/parsers/GrokParser.java | 156 ++---
.../apache/metron/parsers/bolt/ParserBolt.java | 57 +++--
.../parsers/interfaces/MessageParser.java | 28 ++-
.../parsers/interfaces/MessageParserResult.java | 48
.../parsers/websphere/GrokWebSphereParser.java | 217 +--
.../apache/metron/parsers/GrokParserTest.java | 1 +
.../metron/parsers/MultiLineGrokParserTest.java | 146 +
.../MultiLineWithErrorsGrokParserTest.java | 146 +
.../metron/parsers/SampleGrokParserTest.java| 3 +
.../apache/metron/parsers/SquidParserTest.java | 2 +
.../apache/metron/parsers/YafParserTest.java| 3 +
.../metron/parsers/bolt/ParserBoltTest.java | 15 +-
.../parsers/integration/ParserDriver.java | 2 +-
.../websphere/GrokWebSphereParserTest.java | 13 --
.../test/resources/logData/multi_elb_log.txt| 10 +
.../logData/multi_elb_with_errors_log.txt | 13 ++
20 files changed, 824 insertions(+), 180 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/3d923cde/metron-platform/metron-integration-test/src/main/sample/patterns/test
--
diff --git
a/metron-platform/metron-integration-test/src/main/sample/patterns/test
b/metron-platform/metron-integration-test/src/main/sample/patterns/test
index a88a255..ebbf9c4 100644
--- a/metron-platform/metron-integration-test/src/main/sample/patterns/test
+++ b/metron-platform/metron-integration-test/src/main/sample/patterns/test
@@ -1,2 +1,3 @@
YAF_TIME_FORMAT %{YEAR:UNWANTED}-%{MONTHNUM:UNWANTED}-%{MONTHDAY:UNWANTED}[T
]%{HOUR:UNWANTED}:%{MINUTE:UNWANTED}:%{SECOND:UNWANTED}
YAF_DELIMITED
%{NUMBER:start_time}\|%{YAF_TIME_FORMAT:end_time}\|%{SPACE:UNWANTED}%{BASE10NUM:duration}\|%{SPACE:UNWANTED}%{BASE10NUM:rtt}\|%{SPACE:UNWANTED}%{INT:protocol}\|%{SPACE:UNWANTED}%{IP:ip_src_addr}\|%{SPACE:UNWANTED}%{INT:ip_src_port}\|%{SPACE:UNWANTED}%{IP:ip_dst_addr}\|%{SPACE:UNWANTED}%{INT:ip_dst_port}\|%{SPACE:UNWANTED}%{DATA:iflags}\|%{SPACE:UNWANTED}%{DATA:uflags}\|%{SPACE:UNWANTED}%{DATA:riflags}\|%{SPACE:UNWANTED}%{DATA:ruflags}\|%{SPACE:UNWANTED}%{WORD:isn}\|%{SPACE:UNWANTED}%{DATA:risn}\|%{SPACE:UNWANTED}%{DATA:tag}\|%{GREEDYDATA:rtag}\|%{SPACE:UNWANTED}%{INT:pkt}\|%{SPACE:UNWANTED}%{INT:oct}\|%{SPACE:UNWANTED}%{INT:rpkt}\|%{SPACE:UNWANTED}%{INT:roct}\|%{SPACE:UNWANTED}%{INT:app}\|%{GREEDYDATA:end_reason}
+ELBACCESSLOGS %{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb}
%{IP:clientip}:%{INT:clientport} (?:(%{IP:backendip}:?:%{INT:backendport})|-)
%{NUMBER:request_processing_time} %{NUMBER:backend_processing_time}
%{NUMBER:response_processing_time} (?:-|%{INT:elb_status_code})
(?:-|%{INT:backend_status_code}) %{INT:received_bytes} %{INT:sent_bytes}
\"(?:-|(?:%{WORD:verb}
%{URIPROTO:proto}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST:urihost})?(?:%{URIPATH:path}(?:%{URIPARAM:params})?)?(?:
HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest}))\"
\"(?:-|%{DATA:user_agent})\" (?:-|%{NOTSPACE:ssl_cipher})
(?:-|%{NOTSPACE:ssl_protocol})
http://git-wip-us.apache.org/repos/asf/metron/blob/3d923cde/metron-platform/metron-parsers/README.md
--
diff --git a/metron-platform/metron-parsers/README.md
b/metron-platform/metron-parsers/README.md
index 381b0ee..cfcf6ed 100644
--- a/metron-platform/metron-parsers/README.md
+++ b/metron-platform/metron-parsers/README.md
@@ -29,10 +29,12 @@ There are two general types types of parsers:
* Grok parser: `org.apache.metron.parsers.GrokParser` with possible
`parserConfig` entries of
* `grokPath` : The path in HDFS (or in the Jar) to the grok statement
* `patternLabel` : The pattern label to use from the grok statement
+* `multiLine` : The raw data passed in should be handled as a long with
multiple lines, with each line to be parsed
metron git commit: METRON-1813 Stellar REPL Not Initialized with Client JAAS (nickwallen) closes apache/metron#1232
Repository: metron
Updated Branches:
refs/heads/master 1f1b9cdd3 -> 060d17e99
METRON-1813 Stellar REPL Not Initialized with Client JAAS (nickwallen) closes
apache/metron#1232
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/060d17e9
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/060d17e9
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/060d17e9
Branch: refs/heads/master
Commit: 060d17e992a73bc5b65bb9a52d044a3f672ba2c1
Parents: 1f1b9cd
Author: nickwallen
Authored: Wed Oct 10 09:54:21 2018 -0400
Committer: nickallen
Committed: Wed Oct 10 09:54:21 2018 -0400
--
metron-platform/metron-common/src/main/scripts/stellar | 11 ---
1 file changed, 8 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/060d17e9/metron-platform/metron-common/src/main/scripts/stellar
--
diff --git a/metron-platform/metron-common/src/main/scripts/stellar
b/metron-platform/metron-common/src/main/scripts/stellar
index c831c62..34097f2 100644
--- a/metron-platform/metron-common/src/main/scripts/stellar
+++ b/metron-platform/metron-common/src/main/scripts/stellar
@@ -28,10 +28,15 @@ elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ];
then
. /usr/lib/bigtop-utils/bigtop-detect-javahome
fi
+export METRON_SYSCONFIG="/etc/default/metron"
+if [ -f "$METRON_SYSCONFIG" ]; then
+ source $METRON_SYSCONFIG
+fi
+
+export METRON_VERSION="${METRON_VERSION:-${project.version}}"
+export METRON_HOME="${METRON_HOME:-/usr/metron/$METRON_VERSION}"
export HBASE_CONFIGS=$(hbase classpath)
-export METRON_VERSION=${project.version}
-export METRON_HOME=/usr/metron/$METRON_VERSION
export STELLAR_LIB=$(find $METRON_HOME/lib/ -name metron-parsers*.jar)
export MANAGEMENT_LIB=$(find $METRON_HOME/lib/ -name metron-management*.jar)
export PROFILER_LIB=$(find $METRON_HOME/lib/ -name metron-profiler-repl*.jar)
-java $JVMFLAGS -cp
"${CONTRIB:-$METRON_HOME/contrib/*}:$STELLAR_LIB:$MANAGEMENT_LIB:$PROFILER_LIB:$HBASE_CONFIGS"
org.apache.metron.stellar.common.shell.cli.StellarShell "$@"
+java $METRON_JVMFLAGS -cp
"${CONTRIB:-$METRON_HOME/contrib/*}:$STELLAR_LIB:$MANAGEMENT_LIB:$PROFILER_LIB:$HBASE_CONFIGS"
org.apache.metron.stellar.common.shell.cli.StellarShell "$@"
