[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
Go for it, man.  It's all yours. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
@ottobackwards Sure, why don't you start a DISCUSS thread and lay out how 
you envision that looking?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
I can't be assigned jiras but I'll take a stab at METRON-502, unless you 
have already completed it ;)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
I would like to get the discussion on that going, as adding in new models 
is something near and dear.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
That JIRA about making the ParserIntegrationTest better is at 
https://issues.apache.org/jira/browse/METRON-502


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
Yeah, I agree.  I'd like to have a sensor-specific validator as well as a 
global validator.  Right now we have a `MessageFilter`, but it's slightly 
different in intent.  It is intended to pick out which messages we want to pass 
through the parser and drop those that don't meet the criteria.  I'd like to 
see all of these concepts merged into a mapping between stellar statements and 
actions (i.e. DROP, REDIRECT_TO_INVALID, REDIRECT_TO_ERROR) and have sensor 
specific as well as global versions of this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
Yep, I will. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
Are you going to open a jira for that?

-- 

On October 13, 2016 at 10:16:13, Casey Stella (notificati...@github.com)
wrote:

@nickwallen  Funny you mention that; yes, I
think we can. We should modify ParserIntegrationTest here


to check the invalid and error queues and dump their contents and fail
immediately.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub

,
or mute the thread


.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
Ok - from what I can see things are written to the kafka topic, but the 
bolt is never called.  I don't see an crashes in the field validation code at 
this time.  Can anyone make sense of these exceptions I get ( I am only running 
the BRO Integration test )



[brointegrationexception.txt](https://github.com/apache/incubator-metron/files/527323/brointegrationexception.txt)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
@nickwallen Funny you mention that; yes, I think we can.  We should modify 
`ParserIntegrationTest` 
[here](https://github.com/apache/incubator-metron/blob/4a4cb8b117dbb66bbfb4915bca9d871a06682c28/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserIntegrationTest.java#L77)
 to check the invalid and error queues and dump their contents and fail 
immediately.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
I figured out what was going on with this PR and it's kinda interesting, so 
forgive the long-winded explanation.

@ottobackwards is indeed right, it is not apparent why his PR about a 
FieldTransformation and Stellar Function would affect the `YafIntegrationTest`. 
 Unfortunately, it became apparent when it consistently failed on this PR in 
both travis and locally that something was happening.

It turns out the problem is 
[here](https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-integration-test/src/main/config/zookeeper/global.json#L14).
  The keen observer will notice in the global config we are ensuring that the 
`src_ip_addr` and `dst_ip_addr` are valid IP addresses for the integration 
tests.  This, of course, is a typo (that I made :P ).  The correct field names 
are `ip_src_addr` and `ip_dst_addr`.  
Now, you may ask:
* How has this ever worked?
* Given that it did work before, why is it broken now?

Well, for the first question, it was never correct in the integration 
tests, but since we did not exercise the situation where we send in invalid IP 
addresses.  Furthermore, the default behavior for fields which do not exist is 
to return `true` (probably wrong).

For the second question, @ottobackwards has changed the default behavior 
for fields which do not resolve is to return `false` (almost assuredly right).

For clarity to show how I go about debugging these problems, I'll be rather 
explicit in how I figured out what was happening:
* In my git branch, I added @ottobackwards 's repo as a remote repo and 
checked out the branch:
  * `git remote add otto 
https://github.com/ottobackwards/incubator-metron.git`
  * `git checkout otto/METRON-424`
  * `git branch METRON-424-debug`
  * `git checkout METRON-424-debug`
* In Intellij, I ran the YafIntegrationTest to verify that it continued to 
fail
* I set breakpoints at the exit points of the `ParserBolt`:
  * [Line 
135](https://github.com/apache/incubator-metron/blob/4a4cb8b117dbb66bbfb4915bca9d871a06682c28/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java#L135)
  * [Line 
147](https://github.com/apache/incubator-metron/blob/4a4cb8b117dbb66bbfb4915bca9d871a06682c28/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java#L147)
  * [Line 
131](https://github.com/apache/incubator-metron/blob/4a4cb8b117dbb66bbfb4915bca9d871a06682c28/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java#L131)
* It became apparent that we were exiting because the message was not 
valid.  I then set breakpoints inside of @ottobackwards 's code and validated 
that it was doing the right thing, but the fields were coming in as `null`, 
which means the fields were not in the message.
* I tracked the `global.json` down and looked at the field names and there 
the bug became apparent.

I submitted a PR to your PR branch, @ottobackwards, so if you merge that 
in, the build *should* work (fingers crossed ;)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
I'm running as in 
https://maven.apache.org/surefire/maven-surefire-plugin/examples/debugging.html 
and attaching with intellij in test, as you describe actually.  It was how to 
kick off the test in debug that got me


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
I would run one of the tests and put a break point at 
`ParserIntegrationTest.test()` and start walking through it from there.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
Ok - I'm running the integration-test now.  Is there a way to debug it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
FWIW - I am also getting the integration test failures for this PR when I 
run them locally.  Not sure what the problem is though.  I think an exception 
is being swallowed somewhere.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
So, the problem with the tests is this:
```
test(org.apache.metron.parsers.integration.YafIntegrationTest)  Time 
elapsed: 58.442 sec  <<< ERROR!
java.lang.RuntimeException: Too many retries: 11
at 
org.apache.metron.integration.ComponentRunner.process(ComponentRunner.java:140)
at 
org.apache.metron.parsers.integration.ParserIntegrationTest.test(ParserIntegrationTest.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)
at 
org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:98)
at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79)
at 
org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:87)
at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:77)
at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)
at 
org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:88)
at 
org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:51)
at 
org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:44)
at 
org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:27)
at 
org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:37)
at 
org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:42)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
```

This test failed because of a timeout (probably because the box running it 
is overloaded.  The stream of failures after that point is due to the Parser 
Integration tests not shutting down the infrastructure it spins up (e.g. storm, 
etc.) when the tests fail.  I'd close and immediately reopen this PR and it 
should trigger a test rerun.  It is probably worth while making a JIRA to 
correct the integration tests to be smarter and not spew horrible things.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
OK, good then.  I don't have to change anything.
I have no idea why the travis build is failing - it doesn't look todo with 
my changes ( btw there are always so many errors it is hard to look at the log 
).  How does on kick off a new build in the event of a transient error?  Can 
anyone help with what is going on in the log?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #303: METRON-424 ability to validate ip addresses aga...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/incubator-metron/pull/303
  
I would personally vote for `"type" : ["IPV4","IPV6"]`.  I usually push for 
explicit and obvious so something like `retrieve_json.py | grep IPV4` would 
work.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---