[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2017-04-18 Thread merrimanr
Github user merrimanr closed the pull request at:

https://github.com/apache/incubator-metron/pull/308


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-19 Thread mattf-horton
Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r84110876
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
 ---
@@ -181,7 +185,8 @@ public void declareOutputFields(OutputFieldsDeclarer 
declarer) {
   @Override
   public void updateConfig(String path, byte[] data) throws IOException {
 super.updateConfig(path, data);
-if (path.startsWith(ConfigurationType.PARSER.getZookeeperRoot() + "/" 
+ getSensorType())) {
+String pathWithoutTrailingSlash = path.replaceAll("/+$", "");
+if 
(pathWithoutTrailingSlash.equals(ConfigurationType.PARSER.getZookeeperRoot() + 
"/" + getSensorType())) {
--- End diff --

Ok, if sensorType is known to always be a leaf node in ZK, that's 
sufficient.  Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-18 Thread merrimanr
Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83984954
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
 ---
@@ -181,7 +185,8 @@ public void declareOutputFields(OutputFieldsDeclarer 
declarer) {
   @Override
   public void updateConfig(String path, byte[] data) throws IOException {
 super.updateConfig(path, data);
-if (path.startsWith(ConfigurationType.PARSER.getZookeeperRoot() + "/" 
+ getSensorType())) {
+String pathWithoutTrailingSlash = path.replaceAll("/+$", "");
+if 
(pathWithoutTrailingSlash.equals(ConfigurationType.PARSER.getZookeeperRoot() + 
"/" + getSensorType())) {
--- End diff --

I'm sorry I don't follow.  ConfigurationType.PARSER.getZookeeperRoot() is 
constant and sensorType is a leaf node in Zookeeper.  Maybe an example will 
help.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-18 Thread mattf-horton
Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83980978
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
 ---
@@ -181,7 +185,8 @@ public void declareOutputFields(OutputFieldsDeclarer 
declarer) {
   @Override
   public void updateConfig(String path, byte[] data) throws IOException {
 super.updateConfig(path, data);
-if (path.startsWith(ConfigurationType.PARSER.getZookeeperRoot() + "/" 
+ getSensorType())) {
+String pathWithoutTrailingSlash = path.replaceAll("/+$", "");
+if 
(pathWithoutTrailingSlash.equals(ConfigurationType.PARSER.getZookeeperRoot() + 
"/" + getSensorType())) {
--- End diff --

I'm concerned that there may be at least one segment of structured name 
between sensorType and a value that may have changed, thus still requiring a 
"starts with"-like calculation.  How about replace both lines 188 and 189 with:
if (path.matches("^" + ConfigurationType.PARSER.getZookeeperRoot() + "/" + 
getSensorType() + "(/|$)")) {



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-17 Thread merrimanr
Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83749245
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
 ---
@@ -168,4 +177,12 @@ public void declareOutputFields(OutputFieldsDeclarer 
declarer) {
 declarer.declareStream(Constants.INVALID_STREAM, new 
Fields("message"));
 declarer.declareStream(Constants.ERROR_STREAM, new Fields("message"));
   }
+
+  @Override
+  public void updateConfig(String path, byte[] data) throws IOException {
+super.updateConfig(path, data);
+if (path.startsWith(ConfigurationType.PARSER.getZookeeperRoot() + "/" 
+ getSensorType())) {
--- End diff --

Yes good catch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-17 Thread merrimanr
Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83749198
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
 ---
@@ -116,6 +122,9 @@ public void execute(Tuple tuple) {
   boolean ackTuple = !writer.handleAck();
   int numWritten = 0;
   if(sensorParserConfig != null) {
+if (configUpdatedFlag.getAndSet(false)) {
+  
parser.configurationUpdated(getSensorParserConfig().getParserConfig());
--- End diff --

You're right that's definitely incorrect.  I think it would be safer if we 
use the config object fetched in line 118 though.  Is there a benefit to 
updating the config in the middle of the execute method?  We're already 
delaying the change in the parser until the next execute.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-14 Thread mattf-horton
Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83513633
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
 ---
@@ -34,15 +34,15 @@
* @param rawMessage
* @return If null is returned, this is treated as an empty list.
*/
-  List parse(byte[] rawMessage);
+  List parse(byte[] rawMessage, SensorParserConfig sensorParserConfig);
--- End diff --

Of course one really should use AtomicBoolean.  Then, again, no locking.  
Sigh.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-14 Thread mattf-horton
Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83505093
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
 ---
@@ -34,15 +34,15 @@
* @param rawMessage
* @return If null is returned, this is treated as an empty list.
*/
-  List parse(byte[] rawMessage);
+  List parse(byte[] rawMessage, SensorParserConfig sensorParserConfig);
--- End diff --

My inclination (and my original suggestion) is to keep all the asynchronous 
behavior entirely in the ParserBolt (where it already is) and only call 
parser.configurationUpdate() synchronously in the execute method.  So yes to 
that.  Then no synch at all needed in GrokParser.

However, your understanding that you could use ParserBolt#updateConfig() to 
capture notification of changes WITHOUT expensive checks, is brilliant, and 
exactly what updateConfig() is for.  If we do it in ParserBolt, all you need is 
one synchronized routine:
private boolean configUpdatedFlag = false;
synchronized public boolean atomicCheckAndSet(boolean newValue) {
boolean oldValue = configUpdatedFlag; configUpdatedFlag = newValue; return 
oldValue;}

Trivial, safe as can be, and ultra cheap.  No way to get deadlock, altho if 
paranoid, one could give it its own lock object instead of using implicit 
locking on the ParserBolt instance.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-14 Thread merrimanr
Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83497449
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
 ---
@@ -34,15 +34,15 @@
* @param rawMessage
* @return If null is returned, this is treated as an empty list.
*/
-  List parse(byte[] rawMessage);
+  List parse(byte[] rawMessage, SensorParserConfig sensorParserConfig);
--- End diff --

I'll be honest, the Java 8 defaults weren't obvious to me either until I 
looked closer at the MessageParser interface :)

All this synchronization talk makes me nervous.  Would it make things 
simpler if we detected the config change in ParserBolt.execute and 
re-initialized the grok object from there?  We could save a cached copy of the 
config in the ParserBolt on each call to execute and detect changes before 
parse is called using the hashing approach you suggested earlier.  This is the 
reason I initially changed the interface parse method to include the most 
recent config, to avoid multi-threading complexity.  This is approach is very 
similar, we're just using a different method to pass this config in and 
detecting a change in ParserBolt instead of GrokParser.  The parse method is 
synchronous by nature anyways.  It doesn't need to be aware of a config change 
immediately, only when the parse method is called.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-14 Thread mattf-horton
Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83489303
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
 ---
@@ -34,15 +34,15 @@
* @param rawMessage
* @return If null is returned, this is treated as an empty list.
*/
-  List parse(byte[] rawMessage);
+  List parse(byte[] rawMessage, SensorParserConfig sensorParserConfig);
--- End diff --

Hah, didn't think about Java 8 default methods in interfaces.  Of course, 
you're right.

Your above last set of changes looks excellent, but in GrokParser the 
methods configurationUpdate() and init() have to become synchronized methods.  
Then I think the code as written allows the configurationUpdated flag to not 
need further locking, altho different ordering of things might require 
synchronized get/set/check methods.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-14 Thread merrimanr
Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83484585
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
 ---
@@ -34,15 +34,15 @@
* @param rawMessage
* @return If null is returned, this is treated as an empty list.
*/
-  List parse(byte[] rawMessage);
+  List parse(byte[] rawMessage, SensorParserConfig sensorParserConfig);
--- End diff --

I disagree.  The new method is prefixed by "default" meaning MessageParser 
implementations don't have to override that method and the call is a NOOP by 
default.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-14 Thread merrimanr
Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83483522
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/GrokParser.java
 ---
@@ -150,8 +149,8 @@ public void init() {
 
   if (message.size() == 0)
 throw new RuntimeException("Grok statement produced a null 
message. Original message was: "
-+ originalMessage + " and the parsed message was: " + 
message + " . Check the pattern at: "
-+ grokPath);
++ originalMessage + " , parsed message was: " + message + 
" , pattern was: "
++ grokPattern);
--- End diff --

This is exactly what I had in mind.  Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-14 Thread mattf-horton
Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83480785
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
 ---
@@ -34,15 +34,15 @@
* @param rawMessage
* @return If null is returned, this is treated as an empty list.
*/
-  List parse(byte[] rawMessage);
+  List parse(byte[] rawMessage, SensorParserConfig sensorParserConfig);
--- End diff --

+1, with the following caveats:

1. Since parser.configure() takes a (Map parserConfig) 
argument, so should parser.configurationChanged(), rather than a 
sensorParserConfig.  I would also name it parser.configurationUpdate() rather 
than parser.configurationChanged(), but that's obviously minor.

2. Very nice hooking into updateConfig(), BUT we can't call 
parser.configurationUpdate() asynchronously.  Weird things might happen if it's 
in the middle of a message.  Instead, please set a local flag asynchronously in 
ParserBolt, then at each message check-and-clear the flag, and conditionally 
call parser.configurationUpdate() before parser.parseOptional().


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-14 Thread mattf-horton
Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83462598
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/GrokParser.java
 ---
@@ -150,8 +149,8 @@ public void init() {
 
   if (message.size() == 0)
 throw new RuntimeException("Grok statement produced a null 
message. Original message was: "
-+ originalMessage + " and the parsed message was: " + 
message + " . Check the pattern at: "
-+ grokPath);
++ originalMessage + " , parsed message was: " + message + 
" , pattern was: "
++ grokPattern);
--- End diff --

Instead of "+ grokPattern", would you consider:
+ (LOG.isDebugEnabled() ? grokPattern : (patternLabel + " (Turn on DEBUG 
logging to see pattern text.)"))

But if you think it's really important to log the full pattern regardless 
of DEBUG setting, I can accept that.  It's just that under load in the field 
you might run 100,000's of messages through a faulty grokPattern before 
realizing the error, and blow out your log storage.  Which irritates users :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-14 Thread merrimanr
Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83449671
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/GrokParser.java
 ---
@@ -150,8 +149,8 @@ public void init() {
 
   if (message.size() == 0)
 throw new RuntimeException("Grok statement produced a null 
message. Original message was: "
-+ originalMessage + " and the parsed message was: " + 
message + " . Check the pattern at: "
-+ grokPath);
++ originalMessage + " , parsed message was: " + message + 
" , pattern was: "
++ grokPattern);
--- End diff --

You have a point here.  The whole pattern shouldn't be logged on every 
failure, especially since a bad grok pattern will get logged for every message. 
 However there is a reason I did it that way.  As a developer, I prefer to have 
everything I need to troubleshoot something in the same log line.  If we log a 
pattern only when it changes it will be hard to tell which grokPattern caused a 
failure.  You have the message but you'll have to go digging in the logs to 
find the grokPattern that was applied.  Can you think of a way to log 
efficiently but still provide the context needed to troubleshoot an issue?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-14 Thread cestella
Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83447207
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
 ---
@@ -34,15 +34,15 @@
* @param rawMessage
* @return If null is returned, this is treated as an empty list.
*/
-  List parse(byte[] rawMessage);
+  List parse(byte[] rawMessage, SensorParserConfig sensorParserConfig);
--- End diff --

@nickwallen The differences between `configurationChanged` and `configure` 
is that `configure` has an existing intent and this would change it, we would 
need to ensure every parser's configure implementation is compatible with this 
change in intent and doesn't cause regression.

Regarding the similarity in names, I obviously don't think they're too 
similar (otherwise I'd not have suggested it ;).  I think 
`configurationChanged` sounds like a hook that is called when configurations 
change, which is exactly what it would be.  `configure` has an existing 
understanding, which is a one-time configuration.

I've expressed a desire to get rid of one of `init` or `configure`, but I 
don't really care which. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-14 Thread merrimanr
Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83447010
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
 ---
@@ -34,15 +34,15 @@
* @param rawMessage
* @return If null is returned, this is treated as an empty list.
*/
-  List parse(byte[] rawMessage);
+  List parse(byte[] rawMessage, SensorParserConfig sensorParserConfig);
--- End diff --

Before we go any further I think we need to decide if changing the 
MessageParser interface is acceptable or not.  If I remember correctly the 
change to the interface is what prompted this whole conversation.

My opinion is that we quit worrying so much about the interface changing 
and instead focus on getting it right.  Thoughts?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-14 Thread nickwallen
Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83444303
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
 ---
@@ -34,15 +34,15 @@
* @param rawMessage
* @return If null is returned, this is treated as an empty list.
*/
-  List parse(byte[] rawMessage);
+  List parse(byte[] rawMessage, SensorParserConfig sensorParserConfig);
--- End diff --

Also, while we are on the topic, why do we have `parse` and 
`parseOptional`?  Can't we just pick one and go with it instead of muddying up 
the interface with two similar methods?  Or is there something I'm missing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-14 Thread cestella
Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83432385
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
 ---
@@ -34,15 +34,15 @@
* @param rawMessage
* @return If null is returned, this is treated as an empty list.
*/
-  List parse(byte[] rawMessage);
+  List parse(byte[] rawMessage, SensorParserConfig sensorParserConfig);
--- End diff --

Now that I think about it, you could actually do the following:
* Provide a `configurationChanged` method in `MessageParser` with a default 
noop implementation
* Change the argument to `parseOptionally` to take a Config and in the 
default implementation of `parseOptionally` call `configurationChanged` upon 
configuration change.  

This would leave `parse` alone and provide the capability to reconfigure on 
configuration changes seamlessly to parsers going forward.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-14 Thread cestella
Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83431135
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
 ---
@@ -34,15 +34,15 @@
* @param rawMessage
* @return If null is returned, this is treated as an empty list.
*/
-  List parse(byte[] rawMessage);
+  List parse(byte[] rawMessage, SensorParserConfig sensorParserConfig);
--- End diff --

@mattf-horton IMO I do not think we want to check for changes every N 
minutes considering we get push notifications from Zookeeper on configuration 
change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-14 Thread cestella
Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83429210
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
 ---
@@ -34,15 +34,15 @@
* @param rawMessage
* @return If null is returned, this is treated as an empty list.
*/
-  List parse(byte[] rawMessage);
+  List parse(byte[] rawMessage, SensorParserConfig sensorParserConfig);
--- End diff --

I believe that we should use polymorphism to provide `parse(byte[], 
SensorParserConfig)` with a default implementation for `parse(byte[])`.  I 
think that should be done in the `MessageParser` interface using the `default` 
keyword.  We have done precisely this in that interface for `parseOptional`.  
This would provide people the ability to override either one without negatively 
affecting existing parsers.

As for calling configure per message, this would change the semantics of 
the configure method for existing parsers (i.e. shifting from configure being a 
one-time thing to a per-message thing might change how people use the method 
regarding possibly computationally intensive setup).  

As a side-note, I think we should dump the `init()` method from that 
interface and just adopt `configure`.  It's confusing having two setup methods 
with precisely the same semantic characteristics.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-14 Thread nickwallen
Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83418757
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
 ---
@@ -34,15 +34,15 @@
* @param rawMessage
* @return If null is returned, this is treated as an empty list.
*/
-  List parse(byte[] rawMessage);
+  List parse(byte[] rawMessage, SensorParserConfig sensorParserConfig);
--- End diff --

I think our current approach to configuration has worked so far, but as 
we've grown, it has caused some problems like tight coupling and limiting 
reuse.  

@mattf-horton If you (or anyone else) have thoughts on an alternative 
architectural approach for configuration, please share on the dev mailing list. 
 I think we need to do better there.  The core problem to tackle with 
configuration is that components need to be notified of changes made on-the-fly 
and handle them seamlessly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-13 Thread merrimanr
Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83315119
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
 ---
@@ -34,15 +34,15 @@
* @param rawMessage
* @return If null is returned, this is treated as an empty list.
*/
-  List parse(byte[] rawMessage);
+  List parse(byte[] rawMessage, SensorParserConfig sensorParserConfig);
--- End diff --

Where will GrokParser fetch the latest grokPattern from?  We can't cache it 
in the GrokParser object because that copy will be stale as soon as ZK is 
updated.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-13 Thread merrimanr
Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83310911
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
 ---
@@ -34,15 +34,15 @@
* @param rawMessage
* @return If null is returned, this is treated as an empty list.
*/
-  List parse(byte[] rawMessage);
+  List parse(byte[] rawMessage, SensorParserConfig sensorParserConfig);
--- End diff --

I like this idea but I'm a little confused on the implementation.  Wouldn't 
just adding this do the trick:  "BasicParser would provide an overridable 
implementation of the two-argument form as a trivial call to the one-argument 
form"?  Why would we want both methods in the interface?  Am I missing 
something?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-13 Thread merrimanr
Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83303723
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
 ---
@@ -117,7 +117,7 @@ public void execute(Tuple tuple) {
   int numWritten = 0;
   if(sensorParserConfig != null) {
 List fieldValidations = 
getConfigurations().getFieldValidations();
-Optional messages = 
parser.parseOptional(originalMessage);
+Optional messages = 
parser.parseOptional(originalMessage, sensorParserConfig);
--- End diff --

The sensorParserConfig refresh happens asynchronously.   


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-13 Thread merrimanr
Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83301389
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/GrokParser.java
 ---
@@ -132,8 +130,9 @@ public void init() {
 
   @SuppressWarnings("unchecked")
   @Override
-  public List parse(byte[] rawMessage) {
-if (grok == null) {
+  public List parse(byte[] rawMessage, SensorParserConfig 
sensorParserConfig) {
+if (grok == null || isGrokPatternUpdated(sensorParserConfig) || 
isPatternLabelUpdated(sensorParserConfig)) {
+  configure(sensorParserConfig.getParserConfig());
--- End diff --

The timestampField is not what you think it is.  It represents the primary 
timestamp field of the message that's formatted downstream in the parser, not 
the last time the pattern was changed.  I would like to see it go away 
eventually, since you can now do the same thing with Stellar in the 
transformation phase.

I agree with you though, since it's called for every message that check 
needs to be as fast as possible (although you wouldn't use a GrokParser for a 
high velocity sensor anyways).  I think a hash of grokPattern + patternLabel 
will do the trick.  I will add that in.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-13 Thread mattf-horton
Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83283132
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/interfaces/MessageParser.java
 ---
@@ -34,15 +34,15 @@
* @param rawMessage
* @return If null is returned, this is treated as an empty list.
*/
-  List parse(byte[] rawMessage);
+  List parse(byte[] rawMessage, SensorParserConfig sensorParserConfig);
--- End diff --

Currently only the GrokParser wants to see the sensorParserConfig, right?  
Then this patch could be significantly simplified, if desired, by providing 
both one and two-argument polymorphisms of the parse() method.  BasicParser 
would provide an overridable implementation of the two-argument form as a 
trivial call to the one-argument form.  GrokParser can implement the 
one-argument form with either an exception throw, or a two-argument call with 
"current parserConfig" (which would need to be saved from the 
GrokParser:configure() call).  Of course the clients of GrokParser still need 
to be modified, but all the non-Grok parsers and their unit tests no longer 
need to change.  I think that eliminates 20 of the 38 files in this patch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-13 Thread mattf-horton
Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83276073
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/GrokParser.java
 ---
@@ -132,8 +130,9 @@ public void init() {
 
   @SuppressWarnings("unchecked")
   @Override
-  public List parse(byte[] rawMessage) {
-if (grok == null) {
+  public List parse(byte[] rawMessage, SensorParserConfig 
sensorParserConfig) {
+if (grok == null || isGrokPatternUpdated(sensorParserConfig) || 
isPatternLabelUpdated(sensorParserConfig)) {
+  configure(sensorParserConfig.getParserConfig());
--- End diff --

Is the pattern timestampField reliable?  If so, comparing it would be much 
cheaper than comparing the whole grokPattern, which can be long, and 
additionally comparing the patternLabel would be unnecessary.  If not, a SHA1 
hash signature of the grokPattern+patternLabel could be added.  This comparison 
is being done on every message, so worth streamlining.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-13 Thread mattf-horton
Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83277025
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/GrokParser.java
 ---
@@ -150,8 +149,8 @@ public void init() {
 
   if (message.size() == 0)
 throw new RuntimeException("Grok statement produced a null 
message. Original message was: "
-+ originalMessage + " and the parsed message was: " + 
message + " . Check the pattern at: "
-+ grokPath);
++ originalMessage + " , parsed message was: " + message + 
" , pattern was: "
++ grokPattern);
--- End diff --

Including several KB of grokPattern in the exception log might not be 
necessary, especially since if DEBUG is turned on, it was already dumped to log 
in line 106.  Suggest the exception should log only patternLabel and 
timestampField (or patternHash if you add one).  This is comparable to the 
former behavior of logging the path.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-13 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83252938
  
--- Diff: 
metron-platform/metron-parsers/src/main/config/zookeeper/parsers/websphere.json 
---
@@ -3,7 +3,7 @@
   "sensorTopic":"websphere",
   "parserConfig":
   {
-"grokPath":"/patterns/websphere",
+"grokPattern":"# Days - two digit number is used\nDAY \\d{1,2}\n# Time 
- two digit hour, minute, and second\nTIME \\d{2}:\\d{2}:\\d{2}\n# Timestamp - 
month, day, and time\nTIMESTAMP %{MONTH:UNWANTED}\\s+%{DAY:UNWANTED} 
%{TIME:UNWANTED}\n# Generic word field\nWORD \\w+\n# Priority\nPRIORITY \\d+\n# 
Log start - the first part of the log line\nLOGSTART 
<%{PRIORITY:priority}>?%{TIMESTAMP:timestamp_string} %{WORD:hostname}\n# 
Security domain\nSECURITY_DOMAIN [%{WORD:security_domain}]\n# Log middle - the 
middle part of the log line\nLOGMIDDLE 
(\\[%{WORD:security_domain}\\])?\\[%{WORD:event_code}\\]\\[%{WORD:event_type}\\]\\[%{WORD:severity}\\]\n#
 Define IP address formats\nIPV6 
((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)
 
){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:)))(%.+)?\nIPV4
 
(?

[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-13 Thread merrimanr
Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83243880
  
--- Diff: 
metron-platform/metron-parsers/src/main/config/zookeeper/parsers/websphere.json 
---
@@ -3,7 +3,7 @@
   "sensorTopic":"websphere",
   "parserConfig":
   {
-"grokPath":"/patterns/websphere",
+"grokPattern":"# Days - two digit number is used\nDAY \\d{1,2}\n# Time 
- two digit hour, minute, and second\nTIME \\d{2}:\\d{2}:\\d{2}\n# Timestamp - 
month, day, and time\nTIMESTAMP %{MONTH:UNWANTED}\\s+%{DAY:UNWANTED} 
%{TIME:UNWANTED}\n# Generic word field\nWORD \\w+\n# Priority\nPRIORITY \\d+\n# 
Log start - the first part of the log line\nLOGSTART 
<%{PRIORITY:priority}>?%{TIMESTAMP:timestamp_string} %{WORD:hostname}\n# 
Security domain\nSECURITY_DOMAIN [%{WORD:security_domain}]\n# Log middle - the 
middle part of the log line\nLOGMIDDLE 
(\\[%{WORD:security_domain}\\])?\\[%{WORD:event_code}\\]\\[%{WORD:event_type}\\]\\[%{WORD:severity}\\]\n#
 Define IP address formats\nIPV6 
((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)
 
){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:)))(%.+)?\nIPV4
 
(?

[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-13 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83238162
  
--- Diff: 
metron-platform/metron-parsers/src/main/config/zookeeper/parsers/websphere.json 
---
@@ -3,7 +3,7 @@
   "sensorTopic":"websphere",
   "parserConfig":
   {
-"grokPath":"/patterns/websphere",
+"grokPattern":"# Days - two digit number is used\nDAY \\d{1,2}\n# Time 
- two digit hour, minute, and second\nTIME \\d{2}:\\d{2}:\\d{2}\n# Timestamp - 
month, day, and time\nTIMESTAMP %{MONTH:UNWANTED}\\s+%{DAY:UNWANTED} 
%{TIME:UNWANTED}\n# Generic word field\nWORD \\w+\n# Priority\nPRIORITY \\d+\n# 
Log start - the first part of the log line\nLOGSTART 
<%{PRIORITY:priority}>?%{TIMESTAMP:timestamp_string} %{WORD:hostname}\n# 
Security domain\nSECURITY_DOMAIN [%{WORD:security_domain}]\n# Log middle - the 
middle part of the log line\nLOGMIDDLE 
(\\[%{WORD:security_domain}\\])?\\[%{WORD:event_code}\\]\\[%{WORD:event_type}\\]\\[%{WORD:severity}\\]\n#
 Define IP address formats\nIPV6 
((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)
 
){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:)))(%.+)?\nIPV4
 
(?

[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-13 Thread merrimanr
Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83234039
  
--- Diff: 
metron-platform/metron-parsers/src/main/config/zookeeper/parsers/websphere.json 
---
@@ -3,7 +3,7 @@
   "sensorTopic":"websphere",
   "parserConfig":
   {
-"grokPath":"/patterns/websphere",
+"grokPattern":"# Days - two digit number is used\nDAY \\d{1,2}\n# Time 
- two digit hour, minute, and second\nTIME \\d{2}:\\d{2}:\\d{2}\n# Timestamp - 
month, day, and time\nTIMESTAMP %{MONTH:UNWANTED}\\s+%{DAY:UNWANTED} 
%{TIME:UNWANTED}\n# Generic word field\nWORD \\w+\n# Priority\nPRIORITY \\d+\n# 
Log start - the first part of the log line\nLOGSTART 
<%{PRIORITY:priority}>?%{TIMESTAMP:timestamp_string} %{WORD:hostname}\n# 
Security domain\nSECURITY_DOMAIN [%{WORD:security_domain}]\n# Log middle - the 
middle part of the log line\nLOGMIDDLE 
(\\[%{WORD:security_domain}\\])?\\[%{WORD:event_code}\\]\\[%{WORD:event_type}\\]\\[%{WORD:severity}\\]\n#
 Define IP address formats\nIPV6 
((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)
 
){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:)))(%.+)?\nIPV4
 
(?

[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-13 Thread merrimanr
Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83230626
  
--- Diff: 
metron-platform/metron-parsers/src/main/config/zookeeper/parsers/websphere.json 
---
@@ -3,7 +3,7 @@
   "sensorTopic":"websphere",
   "parserConfig":
   {
-"grokPath":"/patterns/websphere",
+"grokPattern":"# Days - two digit number is used\nDAY \\d{1,2}\n# Time 
- two digit hour, minute, and second\nTIME \\d{2}:\\d{2}:\\d{2}\n# Timestamp - 
month, day, and time\nTIMESTAMP %{MONTH:UNWANTED}\\s+%{DAY:UNWANTED} 
%{TIME:UNWANTED}\n# Generic word field\nWORD \\w+\n# Priority\nPRIORITY \\d+\n# 
Log start - the first part of the log line\nLOGSTART 
<%{PRIORITY:priority}>?%{TIMESTAMP:timestamp_string} %{WORD:hostname}\n# 
Security domain\nSECURITY_DOMAIN [%{WORD:security_domain}]\n# Log middle - the 
middle part of the log line\nLOGMIDDLE 
(\\[%{WORD:security_domain}\\])?\\[%{WORD:event_code}\\]\\[%{WORD:event_type}\\]\\[%{WORD:severity}\\]\n#
 Define IP address formats\nIPV6 
((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)
 
){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:)))(%.+)?\nIPV4
 
(?

[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-13 Thread merrimanr
Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83227314
  
--- Diff: 
metron-platform/metron-parsers/src/main/config/zookeeper/parsers/websphere.json 
---
@@ -3,7 +3,7 @@
   "sensorTopic":"websphere",
   "parserConfig":
   {
-"grokPath":"/patterns/websphere",
+"grokPattern":"# Days - two digit number is used\nDAY \\d{1,2}\n# Time 
- two digit hour, minute, and second\nTIME \\d{2}:\\d{2}:\\d{2}\n# Timestamp - 
month, day, and time\nTIMESTAMP %{MONTH:UNWANTED}\\s+%{DAY:UNWANTED} 
%{TIME:UNWANTED}\n# Generic word field\nWORD \\w+\n# Priority\nPRIORITY \\d+\n# 
Log start - the first part of the log line\nLOGSTART 
<%{PRIORITY:priority}>?%{TIMESTAMP:timestamp_string} %{WORD:hostname}\n# 
Security domain\nSECURITY_DOMAIN [%{WORD:security_domain}]\n# Log middle - the 
middle part of the log line\nLOGMIDDLE 
(\\[%{WORD:security_domain}\\])?\\[%{WORD:event_code}\\]\\[%{WORD:event_type}\\]\\[%{WORD:severity}\\]\n#
 Define IP address formats\nIPV6 
((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)
 
){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:)))(%.+)?\nIPV4
 
(?

[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-12 Thread james-sirota
Github user james-sirota commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83144872
  
--- Diff: 
metron-platform/metron-parsers/src/main/config/zookeeper/parsers/websphere.json 
---
@@ -3,7 +3,7 @@
   "sensorTopic":"websphere",
   "parserConfig":
   {
-"grokPath":"/patterns/websphere",
+"grokPattern":"# Days - two digit number is used\nDAY \\d{1,2}\n# Time 
- two digit hour, minute, and second\nTIME \\d{2}:\\d{2}:\\d{2}\n# Timestamp - 
month, day, and time\nTIMESTAMP %{MONTH:UNWANTED}\\s+%{DAY:UNWANTED} 
%{TIME:UNWANTED}\n# Generic word field\nWORD \\w+\n# Priority\nPRIORITY \\d+\n# 
Log start - the first part of the log line\nLOGSTART 
<%{PRIORITY:priority}>?%{TIMESTAMP:timestamp_string} %{WORD:hostname}\n# 
Security domain\nSECURITY_DOMAIN [%{WORD:security_domain}]\n# Log middle - the 
middle part of the log line\nLOGMIDDLE 
(\\[%{WORD:security_domain}\\])?\\[%{WORD:event_code}\\]\\[%{WORD:event_type}\\]\\[%{WORD:severity}\\]\n#
 Define IP address formats\nIPV6 
((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)
 
){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:)))(%.+)?\nIPV4
 
(?

Re: [GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-12 Thread zeo...@gmail.com
+1 to the stackoverflow example.  Much more readable.

Jon

On Wed, Oct 12, 2016, 18:22 cestella  wrote:

> Github user cestella commented on a diff in the pull request:
>
>
> https://github.com/apache/incubator-metron/pull/308#discussion_r83111665
>
> --- Diff:
> metron-platform/metron-parsers/src/main/config/zookeeper/parsers/websphere.json
> ---
> @@ -3,7 +3,7 @@
>"sensorTopic":"websphere",
>"parserConfig":
>{
> -"grokPath":"/patterns/websphere",
> +"grokPattern":"# Days - two digit number is used\nDAY \\d{1,2}\n#
> Time - two digit hour, minute, and second\nTIME \\d{2}:\\d{2}:\\d{2}\n#
> Timestamp - month, day, and time\nTIMESTAMP
> %{MONTH:UNWANTED}\\s+%{DAY:UNWANTED} %{TIME:UNWANTED}\n# Generic word
> field\nWORD \\w+\n# Priority\nPRIORITY \\d+\n# Log start - the first part
> of the log line\nLOGSTART
> <%{PRIORITY:priority}>?%{TIMESTAMP:timestamp_string} %{WORD:hostname}\n#
> Security domain\nSECURITY_DOMAIN [%{WORD:security_domain}]\n# Log middle -
> the middle part of the log line\nLOGMIDDLE
> (\\[%{WORD:security_domain}\\])?\\[%{WORD:event_code}\\]\\[%{WORD:event_type}\\]\\[%{WORD:severity}\\]\n#
> Define IP address formats\nIPV6
> ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)
>  
> ){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:)))(%.+)?\nIPV4
> (?  |[0-1]?[0-9]{1,2}))(?![0-9])\nIP (?:%{IPV6:UNWANTED}|%{IPV4:UNWANTED})\n#
> Message - the message body of the log\nMESSAGE .*\n# WebSphere - the entire
> log message\nWEBSPHERE %{LOGSTART:UNWANTED} %{LOGMIDDLE:UNWANTED}
> %{MESSAGE:message}",
> --- End diff --
>
> Can we please do something to allow multiline strings to be a bit
> easier to write?  My vote is accepting lists of strings or strings for
> `grokPattern` and joining them in the case of list a la
> http://stackoverflow.com/a/7744658
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
-- 

Jon

[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-12 Thread cestella
Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83111665
  
--- Diff: 
metron-platform/metron-parsers/src/main/config/zookeeper/parsers/websphere.json 
---
@@ -3,7 +3,7 @@
   "sensorTopic":"websphere",
   "parserConfig":
   {
-"grokPath":"/patterns/websphere",
+"grokPattern":"# Days - two digit number is used\nDAY \\d{1,2}\n# Time 
- two digit hour, minute, and second\nTIME \\d{2}:\\d{2}:\\d{2}\n# Timestamp - 
month, day, and time\nTIMESTAMP %{MONTH:UNWANTED}\\s+%{DAY:UNWANTED} 
%{TIME:UNWANTED}\n# Generic word field\nWORD \\w+\n# Priority\nPRIORITY \\d+\n# 
Log start - the first part of the log line\nLOGSTART 
<%{PRIORITY:priority}>?%{TIMESTAMP:timestamp_string} %{WORD:hostname}\n# 
Security domain\nSECURITY_DOMAIN [%{WORD:security_domain}]\n# Log middle - the 
middle part of the log line\nLOGMIDDLE 
(\\[%{WORD:security_domain}\\])?\\[%{WORD:event_code}\\]\\[%{WORD:event_type}\\]\\[%{WORD:severity}\\]\n#
 Define IP address formats\nIPV6 
((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)
 
){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:)))(%.+)?\nIPV4
 
(?http://stackoverflow.com/a/7744658


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

2016-10-12 Thread cestella
Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/308#discussion_r83110002
  
--- Diff: 
metron-platform/metron-parsers/src/main/config/zookeeper/parsers/websphere.json 
---
@@ -3,7 +3,7 @@
   "sensorTopic":"websphere",
   "parserConfig":
   {
-"grokPath":"/patterns/websphere",
+"grokPattern":"# Days - two digit number is used\nDAY \\d{1,2}\n# Time 
- two digit hour, minute, and second\nTIME \\d{2}:\\d{2}:\\d{2}\n# Timestamp - 
month, day, and time\nTIMESTAMP %{MONTH:UNWANTED}\\s+%{DAY:UNWANTED} 
%{TIME:UNWANTED}\n# Generic word field\nWORD \\w+\n# Priority\nPRIORITY \\d+\n# 
Log start - the first part of the log line\nLOGSTART 
<%{PRIORITY:priority}>?%{TIMESTAMP:timestamp_string} %{WORD:hostname}\n# 
Security domain\nSECURITY_DOMAIN [%{WORD:security_domain}]\n# Log middle - the 
middle part of the log line\nLOGMIDDLE 
(\\[%{WORD:security_domain}\\])?\\[%{WORD:event_code}\\]\\[%{WORD:event_type}\\]\\[%{WORD:severity}\\]\n#
 Define IP address formats\nIPV6 
((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)
 
){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:)))(%.+)?\nIPV4
 
(?