When things change in hdfs, how do we know

[Otto Fowler]

At the moment, when a grok file or something changes in HDFS, how do we
know?  Do we have to restart the parser topology to pick it up?
Just trying to clarify for myself.

ottO

Re: [GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[Otto Fowler]

Maybe we need an adding support for a new platform doc


On January 25, 2018 at 19:37:47, nickwallen (g...@git.apache.org) wrote:

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/903

> @lvets: Just for my understanding, but why Ubuntu Trusty? In April that
will be 2 full Ubuntu LTS versions behind the then current one...

Because that's the requirement that I need to support. All the work around
the DEBs, the Mpack, Ansible setup was driven towards that.

If you or anyone else wants to add support for a newer version that can
also be done, but someone will have to put in the effort to do so.


---

Re: Metron User Community Meeting Call

[Kyle Richardson]

Thanks! I'll be there. Excited to hear Ahmed's successes and challenges.

-Kyle

On Thu, Jan 25, 2018 at 7:44 PM zeo...@gmail.com  wrote:

> Thanks Otto, I'm in to attend at that time/place.
>
> Jon
>
> On Thu, Jan 25, 2018, 14:45 Otto Fowler  wrote:
>
>> I would like to propose a Metron user community meeting. I propose that
>> we set the meeting next week, and will throw out Wednesday, January 31st at
>> 09:30AM PST, 12:30 on the East Coast and 5:30 in London Towne. This meeting
>> will be held over a web-ex, the details of which will be included in the
>> actual meeting notice.
>> Topics
>>
>> We have a volunteer for a community member presentation:
>>
>> Ahmed Shah (PMP, M. Eng.) Cybersecurity Analyst & Developer GCR -
>> Cybersecurity Operations Center Carleton University - cugcr.com
>>
>> Ahmed would like to talk to the community about
>>
>>-
>>
>>Who the GCR group is
>>-
>>
>>How they use Metron 0.4.1
>>-
>>
>>Walk through their dashboards, UI management screen, nifi
>>-
>>
>>Challenges we faced up until now
>>
>> I would like to thank Ahmed for stepping forward for this meeting.
>>
>> If you have something you would like to present or talk about please
>> reply here! Maybe we can have people ask for “A better explanation of
>> feature X” type things?
>> Metron User Community Meetings
>>
>> User Community Meetings are a means for realtime discussion of
>> experiences with Apache Metron, or demonstration of how the community is
>> using or will be using Apache Metron.
>>
>> These meetings are geared towards:
>>
>>-
>>
>>Demonstrations and knowledge sharing as opposed to technical
>>discussion or implementation details from members of the Apache Metron
>>Community
>>-
>>
>>Existing Feature demonstrations
>>-
>>
>>Proposed Feature demonstrations
>>-
>>
>>Community feedback
>>
>> These meetings are *not* for :
>>
>>-
>>
>>Support discussions. Those are best left to the mailing lists.
>>-
>>
>>Development discussions. There is another type of meeting for that.
>>
>>
>>
>>
>
> --
>
> Jon
>

Re: Metron User Community Meeting Call

[zeo...@gmail.com]

Thanks Otto, I'm in to attend at that time/place.

Jon

On Thu, Jan 25, 2018, 14:45 Otto Fowler  wrote:

> I would like to propose a Metron user community meeting. I propose that we
> set the meeting next week, and will throw out Wednesday, January 31st at
> 09:30AM PST, 12:30 on the East Coast and 5:30 in London Towne. This meeting
> will be held over a web-ex, the details of which will be included in the
> actual meeting notice.
> Topics
>
> We have a volunteer for a community member presentation:
>
> Ahmed Shah (PMP, M. Eng.) Cybersecurity Analyst & Developer GCR -
> Cybersecurity Operations Center Carleton University - cugcr.com
>
> Ahmed would like to talk to the community about
>
>-
>
>Who the GCR group is
>-
>
>How they use Metron 0.4.1
>-
>
>Walk through their dashboards, UI management screen, nifi
>-
>
>Challenges we faced up until now
>
> I would like to thank Ahmed for stepping forward for this meeting.
>
> If you have something you would like to present or talk about please reply
> here! Maybe we can have people ask for “A better explanation of feature
> X” type things?
> Metron User Community Meetings
>
> User Community Meetings are a means for realtime discussion of experiences
> with Apache Metron, or demonstration of how the community is using or will
> be using Apache Metron.
>
> These meetings are geared towards:
>
>-
>
>Demonstrations and knowledge sharing as opposed to technical
>discussion or implementation details from members of the Apache Metron
>Community
>-
>
>Existing Feature demonstrations
>-
>
>Proposed Feature demonstrations
>-
>
>Community feedback
>
> These meetings are *not* for :
>
>-
>
>Support discussions. Those are best left to the mailing lists.
>-
>
>Development discussions. There is another type of meeting for that.
>
>
>
>

-- 

Jon

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/903
  
I am liking a combination of the suggestions from @lvets and @cestella.  
Something like this maybe?
* `dev-on-centos6`
* `dev-on-ubuntu14`

I like the name because of points made by others...
* The name identifies these as development environments
* Allows us to support multiple versions of the same platform; Ubuntu 16, 
CentOS 7, etc.


---

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/903
  
> @lvets: Just for my understanding, but why Ubuntu Trusty? In April that 
will be 2 full Ubuntu LTS versions behind the then current one...

Because that's the requirement that I need to support.  All the work around 
the DEBs, the Mpack, Ansible setup was driven towards that.  

If you or anyone else wants to add support for a newer version that can 
also be done, but someone will have to put in the effort to do so.


---

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/903
  
@lvets trusty is 14.04.  As far as I'm aware the only newer LTS is 16.04, 
with a new one expected in April.  https://wiki.ubuntu.com/Releases


---

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[lvets]

Github user lvets commented on the issue:

https://github.com/apache/metron/pull/903
  
- Wouldn't `dev-on-centos` and `dev-on-ubuntu` or even `metron-dev-centos` 
and `metron-dev-ubuntu` make more sense? I don't think we want people to expect 
that this VM is sufficient to run in production?
- Just for my understanding, but why Ubuntu Trusty? In April that will be 2 
full Ubuntu LTS versions behind the then current one...


---

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/903
  
Honestly my only beef with this is the directory naming.  I think I'd 
prefer `metron-deployment/vagrant/{centos_$VERSION,ubuntu_$VERSION}` because 
it's possible that we may want 2 separate versions of centos and the 
`on_metron` is kinda redundant.


---

[GitHub] metron pull request #902: METRON-1413 Add Metron Commit Tool

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/902


---

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/903
  
Thanks @ottobackwards .  I'll see if we can get any more reviewers before I 
merge this.


---

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/903
  
+1 -> ran up both images, everything checked out
Ship it


---

[GitHub] metron issue #901: METRON-1410 [MPACK] Check for existing HBASE tables befor...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/901
  
I'm going to wait for @lvets a chance to try his scenario


---

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/903
  
> @ottobackwards : Do we expect there to be issues with 2.6? Is this PR and 
Casey's 2.6 pr going to conflict or have issues?

Yes, we will need to retest one or the other.  I am open to helping retest 
either this or #907, whichever goes in last.  If you or @cestella see an 
advantage to one or the other going first, I am all ears.

Fortunately, I do not expect there to be a problem.  Portions of this PR 
were previously tested against Ambari 2.6.  But of course, anything can happen, 
so we will need to retest either way.






---

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/903
  
I spun this up again on both Ubuntu and CentOS.  Both worked successfully.  
I am happy with it now @ottobackwards .  Give her another go when you can.  
Thanks.


---

[GitHub] metron pull request #910: METRON-1430: Isolate jackson from being used as ar...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/910#discussion_r163967410
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/IndexDao.java
 ---
@@ -140,9 +147,9 @@ default Document getPatchedDocument(PatchRequest request
 throw new OriginalNotFoundException("Unable to patch an document 
that doesn't exist and isn't specified.");
   }
 }
-JsonNode originalNode = JSONUtils.INSTANCE.convert(latest, 
JsonNode.class);
-JsonNode patched = JSONUtils.INSTANCE.applyPatch(request.getPatch(), 
originalNode);
-Map updated = JSONUtils.INSTANCE.getMapper()
+JsonNode originalNode = _mapper.get().convertValue(latest, 
JsonNode.class);
+JsonNode patched = JsonPatch.apply(request.getPatch(), originalNode);
+Map updated = _mapper.get()
 .convertValue(patched, new TypeReference>() 
{});
--- End diff --

no, I don't think I want to pass in a JsonNode to 
`JSONUtils.INSTANCE.convert()`.  I'm not sure WHAT happens there, honestly in a 
situation where the JsonNode we pass in isn't the jackson version that 
JSONUtils is using in metron-common.  Also, I think I'd just like to keep 
JSONUtils out of this class since it's using Jackson.


---

[GitHub] metron pull request #910: METRON-1430: Isolate jackson from being used as ar...

[mmiklavc]

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/910#discussion_r163962696
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/IndexDao.java
 ---
@@ -140,9 +147,9 @@ default Document getPatchedDocument(PatchRequest request
 throw new OriginalNotFoundException("Unable to patch an document 
that doesn't exist and isn't specified.");
   }
 }
-JsonNode originalNode = JSONUtils.INSTANCE.convert(latest, 
JsonNode.class);
-JsonNode patched = JSONUtils.INSTANCE.applyPatch(request.getPatch(), 
originalNode);
-Map updated = JSONUtils.INSTANCE.getMapper()
+JsonNode originalNode = _mapper.get().convertValue(latest, 
JsonNode.class);
+JsonNode patched = JsonPatch.apply(request.getPatch(), originalNode);
+Map updated = _mapper.get()
 .convertValue(patched, new TypeReference>() 
{});
--- End diff --

JSONUtils.MAP_SUPPLIER here?


---

[GitHub] metron issue #902: METRON-1413 Add Metron Commit Tool

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/902
  
+1


---

[GitHub] metron issue #910: METRON-1430: Isolate jackson from being used as arguments...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/910
  
Yeah, this is the first step to moving it out, I'd say.  This was useful 
independent of #907 because it made some things a lot cleaner (namely fewer 
`new TypeReference>() {}` and more 
`JSONUtils.MAP_SUPPLIER`).


---

[GitHub] metron issue #902: METRON-1413 Add Metron Commit Tool

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/902
  
I merged with master, so I expect Travis to be happy now.  Just need +1s 
and I'll get this in to allow for any follow-ons.


---

[GitHub] metron issue #901: METRON-1410 [MPACK] Check for existing HBASE tables befor...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/901
  
+1 Looks good @ottobackwards.  Thanks for fixing this!

I have not tested this myself, but it looks solid.  Let me know if you'd 
prefer me to spin this up to get a second test run in.  Otherwise, I assume 
that what you've done is sufficient.


---

[GitHub] metron issue #910: METRON-1430: Isolate jackson from being used as arguments...

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/910
  
@ottobackwards I was envisioning a metron-deps project of some sort. 
Jackson and Guava are perennial offenders in the Hadoop stack, and a facade of 
some sort would be useful to insulate us from the ground shifting beneath us.


---

Re: [DISCUSS] Using JSON Path to support more complex documents with the JSONMap Parser

[Otto Fowler]

Sure it helps, but I am not sure I answered __your__ questions?

As I mentioned, we already use

Map rawMap = JSONUtils.INSTANCE.load(originalString, new
TypeReference>() {
});


So, using JSONPath which is using the same object mapper operation under
the covers is not a change.
We were already reading the complete document in.


On January 25, 2018 at 15:06:28, Matt Foley (ma...@apache.org) wrote:

Heh, as I said, I was looking in Python. For SAX-like JSON parsers I found
numerous libraries, most built on top of an underlying Python library named
ijson, which is itself based on a C library called yajl.

The yajl page (http://lloyd.github.io/yajl/ ) lists a double handful of
language bindings but, annoyingly, none for Java; nor does Google seem to
know of any.

In Java, there's a library named json-simple in the Google Code Archive
which claims a SAX-like interface and broad production-level
adoption/robustness: https://code.google.com/archive/p/json-simple/ . I
don't have experience with it.

Of course, the gold standard json library for Java is Jackson. It documents
stream-based parsing, but not "SAX-like".
https://github.com/FasterXML/jackson-docs/wiki/JacksonStreamingApi
indicates that using it is equivalent to writing a parser, which suggests
(disappointingly) somewhat lower-level than SAX api.
http://www.cowtowncoder.com/blog/archives/2009/01/entry_132.html compares
Jackson streaming interface to Stax and SAX, and says it is like Stax
Cursor api, claiming simpler use than SAX (about which I have no opinion).
So I think most people use Jackson for non-streaming consumption of json.

JsonPath implementation uses Jackson under the hood, which seems good to me
-- professionals don't recreate the wheel.
And it has the charm (for this community) of a DSL-like interface. It's
likely a good choice.

Hope this helps,
--Matt

On 1/25/18, 10:05 AM, "Otto Fowler"  wrote:

In other words, I don’t believe the issue is parsing, but rather searching
and extracting.

I have used SAX with xml as well, can you point me to the json equivalent
you found?


On January 25, 2018 at 13:01:58, Otto Fowler (ottobackwa...@gmail.com)
wrote:

JSONPath is indeed what nifi uses. I used their implementation as a guide.
I believe starting with a path would be a good minimum viable, a good
start.
We could support multiple paths of course.

Beside the fact that I knew NiFi used this approach, I believe that
JSONPath provides a flexible mechanism for defining
the targets within the document, and would make this more usable across
various document structures.

We already do full document with simple json btw.

On January 25, 2018 at 12:45:12, Matt Foley (ma...@apache.org) wrote:

Hi Otto,
Oddly, I had reason a couple weeks ago to try to figure out a streaming
parser for very large json objects -- altho it was in Python rather than
Java.
Search showed two basic approaches, both unsurprisingly modeled on xml
processing:
- SAX-like parsing
- XPath-like parsing

Both are capable of true streaming interface, that is one doesn't have to
load the whole json into memory first.
The sound-bite comparison of the two, thanks to stackoverflow, is:

> SAX is a top-down parser and allows serial access to a XML document, and
works well for read only [serial, streamed] access.
> XPath is useful when you only need a couple of values from the XML
document, and you know where to find them (you know the path of the data,
/root/item/challange/text).
> [XPath is] certainly easier to use, ... whereas ... SAX will always be a
lot more awkward to program than XPath.

Having used SAX before, I agree it's got an "awkward" api, but it's quite
usable and does the job.
I haven't been hands-on with XPath.

Is XPath (or rather JSONPath) what NiFi uses?
And is it sufficient for our needs to have a fixed path to the message
sequence in any given json bundle?

Thanks,
--Matt


On 1/25/18, 7:57 AM, "Otto Fowler"  wrote:

While it would be preferred if all data streamed into the parsers is
already in ‘stream’ form, as opposed to ‘batched’ form, it may not always
be possible, or possible at every step of system development.

I was wondering if it would be worth adding optional support to the JSONMap
Parser to support more complex documents, and split them in the parser into
multiple messages. This is similar in function to the JSON Splitter
processor in NiFi

So, a document would come into the JSONMap Parser from Kafka, with some
embedded set of the real message content, such as in this simplified
example:

{
“messages" : [
{ message1},
{ message2},
….
{messageN}
]
}

the JSONMap Parser, would have a new configuration item for message
selection, that would be a JSON Path expression

“messageSelector” : “$.messages “

Inside the JSONMap Parser, it would evaluate the expression, and do the
same processing on each item returned by the expression list.

the Parser interface already supports 

Re: [DISCUSS] Using JSON Path to support more complex documents with the JSONMap Parser

[Matt Foley]

Heh, as I said, I was looking in Python.  For SAX-like JSON parsers I found 
numerous libraries, most built on top of an underlying Python library named 
ijson, which is itself based on a C library called yajl.

The yajl page (http://lloyd.github.io/yajl/ ) lists a double handful of 
language bindings but, annoyingly, none for Java; nor does Google seem to know 
of any.

In Java, there's a library named json-simple in the Google Code Archive which 
claims a SAX-like interface and broad production-level adoption/robustness: 
https://code.google.com/archive/p/json-simple/ .  I don't have experience with 
it.

Of course, the gold standard json library for Java is Jackson.  It documents 
stream-based parsing, but not "SAX-like".
https://github.com/FasterXML/jackson-docs/wiki/JacksonStreamingApi indicates 
that using it is equivalent to writing a parser, which suggests 
(disappointingly) somewhat lower-level than SAX api.
http://www.cowtowncoder.com/blog/archives/2009/01/entry_132.html compares 
Jackson streaming interface to Stax and SAX, and says it is like Stax Cursor 
api, claiming simpler use than SAX (about which I have no opinion).
So I think most people use Jackson for non-streaming consumption of json.

JsonPath implementation uses Jackson under the hood, which seems good to me -- 
professionals don't recreate the wheel.
And it has the charm (for this community) of a DSL-like interface.  It's likely 
a good choice.

Hope this helps,
--Matt

On 1/25/18, 10:05 AM, "Otto Fowler"  wrote:

In other words, I don’t believe the issue is parsing, but rather searching
and extracting.

I have used SAX with xml as well, can you point me to the json equivalent
you found?


On January 25, 2018 at 13:01:58, Otto Fowler (ottobackwa...@gmail.com)
wrote:

JSONPath is indeed what nifi uses.  I used their implementation as a guide.
I believe starting with a path would be a good minimum viable, a good start.
We could support multiple paths of course.

Beside the fact that I knew NiFi used this approach, I believe that
JSONPath provides a flexible mechanism for defining
the targets within the document, and would make this more usable across
various document structures.

We already do full document with simple json btw.

On January 25, 2018 at 12:45:12, Matt Foley (ma...@apache.org) wrote:

Hi Otto,
Oddly, I had reason a couple weeks ago to try to figure out a streaming
parser for very large json objects -- altho it was in Python rather than
Java.
Search showed two basic approaches, both unsurprisingly modeled on xml
processing:
- SAX-like parsing
- XPath-like parsing

Both are capable of true streaming interface, that is one doesn't have to
load the whole json into memory first.
The sound-bite comparison of the two, thanks to stackoverflow, is:

> SAX is a top-down parser and allows serial access to a XML document, and
works well for read only [serial, streamed] access.
> XPath is useful when you only need a couple of values from the XML
document, and you know where to find them (you know the path of the data,
/root/item/challange/text).
> [XPath is] certainly easier to use, ... whereas ... SAX will always be a
lot more awkward to program than XPath.

Having used SAX before, I agree it's got an "awkward" api, but it's quite
usable and does the job.
I haven't been hands-on with XPath.

Is XPath (or rather JSONPath) what NiFi uses?
And is it sufficient for our needs to have a fixed path to the message
sequence in any given json bundle?

Thanks,
--Matt


On 1/25/18, 7:57 AM, "Otto Fowler"  wrote:

While it would be preferred if all data streamed into the parsers is
already in ‘stream’ form, as opposed to ‘batched’ form, it may not always
be possible, or possible at every step of system development.

I was wondering if it would be worth adding optional support to the JSONMap
Parser to support more complex documents, and split them in the parser into
multiple messages. This is similar in function to the JSON Splitter
processor in NiFi

So, a document would come into the JSONMap Parser from Kafka, with some
embedded set of the real message content, such as in this simplified
example:

{
“messages" : [
{ message1},
{ message2},
….
{messageN}
]
}

the JSONMap Parser, would have a new configuration item for message
selection, that would be a JSON Path expression

“messageSelector” : “$.messages “

Inside the JSONMap Parser, it would evaluate the expression, and do the
same processing on each item returned by the expression list.

the Parser interface already supports returning multiple message objects
from a single byte[] input.

[GitHub] metron issue #910: METRON-1430: Isolate jackson from being used as arguments...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/910
  
This looks awesome at first look.  I'm a big fan of doing this regardless 
of the shading issue.  The only thought that comes to mind is that there is a 
tipping point where a *Utils class isn't the right thing, that is it outgrown.  
I feel like we might be there for JSONUtils.

We can do that in another pr however I guess.


---

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/903
  
@nickwallen let me know when you feel ok about it, I'll run it through 
again.


---

[GitHub] metron pull request #910: METRON-1430: Isolate jackson from being used as ar...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/910

METRON-1430: Isolate jackson from being used as arguments or returns from 
JSONUtils

## Contributor Comments
Currently jackson is used as part of our internal API to JSONUtils.  The 
problem here is when we shade and relocate jackson.  Suddenly we can't use 
JSONUtils.  We should avoid operating on jackson primitives and rather provide 
a convenient wrapper around the places where we use jackson as part of the API.

This is required for #907 because this is where we are forced to relocate 
jackson and there is a competing version on the classpath.

Validate by running up vagrant and ensuring data flows through all 
topologies, data shows up in the alerts UI.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron 
jackson_api_isolation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/910.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #910


commit 2f553e85ca8cf41d35395c6e85d70aab7b39db09
Author: cstella 
Date:   2018-01-25T18:59:18Z

Isolate JSONUtils.load from exposing Jackson

commit 9c8b3bd821f8feb2f4d39ca640706ebb7a89dbe6
Author: cstella 
Date:   2018-01-25T19:32:43Z

Updating JSONUtils to avoid exposing jackson




---

Metron User Community Meeting Call

[Otto Fowler]

I would like to propose a Metron user community meeting. I propose that we
set the meeting next week, and will throw out Wednesday, January 31st at
09:30AM PST, 12:30 on the East Coast and 5:30 in London Towne. This meeting
will be held over a web-ex, the details of which will be included in the
actual meeting notice.
Topics

We have a volunteer for a community member presentation:

Ahmed Shah (PMP, M. Eng.) Cybersecurity Analyst & Developer GCR -
Cybersecurity Operations Center Carleton University - cugcr.com

Ahmed would like to talk to the community about

   -

   Who the GCR group is
   -

   How they use Metron 0.4.1
   -

   Walk through their dashboards, UI management screen, nifi
   -

   Challenges we faced up until now

I would like to thank Ahmed for stepping forward for this meeting.

If you have something you would like to present or talk about please reply
here! Maybe we can have people ask for “A better explanation of feature
X” type things?
Metron User Community Meetings

User Community Meetings are a means for realtime discussion of experiences
with Apache Metron, or demonstration of how the community is using or will
be using Apache Metron.

These meetings are geared towards:

   -

   Demonstrations and knowledge sharing as opposed to technical discussion
   or implementation details from members of the Apache Metron Community
   -

   Existing Feature demonstrations
   -

   Proposed Feature demonstrations
   -

   Community feedback

These meetings are *not* for :

   -

   Support discussions. Those are best left to the mailing lists.
   -

   Development discussions. There is another type of meeting for that.

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/903
  
> @ottobackwards : Failure during vagrant up for metron-on-ubuntu

Thanks, Otto. Yep, I messed that up.  I pushed the fix, but I am going to 
run through full CentOS and Ubuntu deployments just to be sure.




---

Re: [DISCUSS] Using JSON Path to support more complex documents with the JSONMap Parser

[Otto Fowler]

In other words, I don’t believe the issue is parsing, but rather searching
and extracting.

I have used SAX with xml as well, can you point me to the json equivalent
you found?


On January 25, 2018 at 13:01:58, Otto Fowler (ottobackwa...@gmail.com)
wrote:

JSONPath is indeed what nifi uses.  I used their implementation as a guide.
I believe starting with a path would be a good minimum viable, a good start.
We could support multiple paths of course.

Beside the fact that I knew NiFi used this approach, I believe that
JSONPath provides a flexible mechanism for defining
the targets within the document, and would make this more usable across
various document structures.

We already do full document with simple json btw.

On January 25, 2018 at 12:45:12, Matt Foley (ma...@apache.org) wrote:

Hi Otto,
Oddly, I had reason a couple weeks ago to try to figure out a streaming
parser for very large json objects -- altho it was in Python rather than
Java.
Search showed two basic approaches, both unsurprisingly modeled on xml
processing:
- SAX-like parsing
- XPath-like parsing

Both are capable of true streaming interface, that is one doesn't have to
load the whole json into memory first.
The sound-bite comparison of the two, thanks to stackoverflow, is:

> SAX is a top-down parser and allows serial access to a XML document, and
works well for read only [serial, streamed] access.
> XPath is useful when you only need a couple of values from the XML
document, and you know where to find them (you know the path of the data,
/root/item/challange/text).
> [XPath is] certainly easier to use, ... whereas ... SAX will always be a
lot more awkward to program than XPath.

Having used SAX before, I agree it's got an "awkward" api, but it's quite
usable and does the job.
I haven't been hands-on with XPath.

Is XPath (or rather JSONPath) what NiFi uses?
And is it sufficient for our needs to have a fixed path to the message
sequence in any given json bundle?

Thanks,
--Matt


On 1/25/18, 7:57 AM, "Otto Fowler"  wrote:

While it would be preferred if all data streamed into the parsers is
already in ‘stream’ form, as opposed to ‘batched’ form, it may not always
be possible, or possible at every step of system development.

I was wondering if it would be worth adding optional support to the JSONMap
Parser to support more complex documents, and split them in the parser into
multiple messages. This is similar in function to the JSON Splitter
processor in NiFi

So, a document would come into the JSONMap Parser from Kafka, with some
embedded set of the real message content, such as in this simplified
example:

{
“messages" : [
{ message1},
{ message2},
….
{messageN}
]
}

the JSONMap Parser, would have a new configuration item for message
selection, that would be a JSON Path expression

“messageSelector” : “$.messages “

Inside the JSONMap Parser, it would evaluate the expression, and do the
same processing on each item returned by the expression list.

the Parser interface already supports returning multiple message objects
from a single byte[] input.

There is a performance penalty to be paid here, and it is more than just
doing more than one message due to the JSONPath evaluation.

I can see this being useful in a couple of circumstances:

-

You want to work with some document format with metron but do not have
NiFi or the equivalent available or setup yet
-

You want to prototype with Metron before you get the ‘preprocessing’
setup
-

You are not going to be able to use NiFi and are ok with the performance

I have something in github to look at for more detail :
ottobackwards/json-path-play


Thoughts?

Re: [DISCUSS] Using JSON Path to support more complex documents with the JSONMap Parser

[Otto Fowler]

JSONPath is indeed what nifi uses.  I used their implementation as a guide.
I believe starting with a path would be a good minimum viable, a good start.
We could support multiple paths of course.

Beside the fact that I knew NiFi used this approach, I believe that
JSONPath provides a flexible mechanism for defining
the targets within the document, and would make this more usable across
various document structures.

We already do full document with simple json btw.

On January 25, 2018 at 12:45:12, Matt Foley (ma...@apache.org) wrote:

Hi Otto,
Oddly, I had reason a couple weeks ago to try to figure out a streaming
parser for very large json objects -- altho it was in Python rather than
Java.
Search showed two basic approaches, both unsurprisingly modeled on xml
processing:
- SAX-like parsing
- XPath-like parsing

Both are capable of true streaming interface, that is one doesn't have to
load the whole json into memory first.
The sound-bite comparison of the two, thanks to stackoverflow, is:

> SAX is a top-down parser and allows serial access to a XML document, and
works well for read only [serial, streamed] access.
> XPath is useful when you only need a couple of values from the XML
document, and you know where to find them (you know the path of the data,
/root/item/challange/text).
> [XPath is] certainly easier to use, ... whereas ... SAX will always be a
lot more awkward to program than XPath.

Having used SAX before, I agree it's got an "awkward" api, but it's quite
usable and does the job.
I haven't been hands-on with XPath.

Is XPath (or rather JSONPath) what NiFi uses?
And is it sufficient for our needs to have a fixed path to the message
sequence in any given json bundle?

Thanks,
--Matt


On 1/25/18, 7:57 AM, "Otto Fowler"  wrote:

While it would be preferred if all data streamed into the parsers is
already in ‘stream’ form, as opposed to ‘batched’ form, it may not always
be possible, or possible at every step of system development.

I was wondering if it would be worth adding optional support to the JSONMap
Parser to support more complex documents, and split them in the parser into
multiple messages. This is similar in function to the JSON Splitter
processor in NiFi

So, a document would come into the JSONMap Parser from Kafka, with some
embedded set of the real message content, such as in this simplified
example:

{
“messages" : [
{ message1},
{ message2},
….
{messageN}
]
}

the JSONMap Parser, would have a new configuration item for message
selection, that would be a JSON Path expression

“messageSelector” : “$.messages “

Inside the JSONMap Parser, it would evaluate the expression, and do the
same processing on each item returned by the expression list.

the Parser interface already supports returning multiple message objects
from a single byte[] input.

There is a performance penalty to be paid here, and it is more than just
doing more than one message due to the JSONPath evaluation.

I can see this being useful in a couple of circumstances:

-

You want to work with some document format with metron but do not have
NiFi or the equivalent available or setup yet
-

You want to prototype with Metron before you get the ‘preprocessing’
setup
-

You are not going to be able to use NiFi and are ok with the performance

I have something in github to look at for more detail :
ottobackwards/json-path-play


Thoughts?

Re: [DISCUSS] Using JSON Path to support more complex documents with the JSONMap Parser

[Matt Foley]

Hi Otto,
Oddly, I had reason a couple weeks ago to try to figure out a streaming parser 
for very large json objects -- altho it was in Python rather than Java.
Search showed two basic approaches, both unsurprisingly modeled on xml 
processing:
- SAX-like parsing
- XPath-like parsing

Both are capable of true streaming interface, that is one doesn't have to load 
the whole json into memory first.
The sound-bite comparison of the two, thanks to stackoverflow, is:

> SAX is a top-down parser and allows serial access to a XML document, and 
> works well for read only [serial, streamed] access. 
> XPath is useful when you only need a couple of values from the XML document, 
> and you know where to find them (you know the path of the data, 
> /root/item/challange/text).
> [XPath is] certainly easier to use, ... whereas ... SAX will always be a lot 
> more awkward to program than XPath.

Having used SAX before, I agree it's got an "awkward" api, but it's quite 
usable and does the job.
I haven't been hands-on with XPath.

Is XPath (or rather JSONPath) what NiFi uses?  
And is it sufficient for our needs to have a fixed path to the message sequence 
in any given json bundle?

Thanks,
--Matt


On 1/25/18, 7:57 AM, "Otto Fowler"  wrote:

While it would be preferred if all data streamed into the parsers is
already in ‘stream’ form, as opposed to ‘batched’ form, it may not always
be possible, or possible at every step of system development.

I was wondering if it would be worth adding optional support to the JSONMap
Parser to support more complex documents, and split them in the parser into
multiple messages. This is similar in function to the JSON Splitter
processor in NiFi

So, a document would come into the JSONMap Parser from Kafka, with some
embedded set of the real message content, such as in this simplified
example:

{
“messages" : [
{ message1},
{ message2},
….
{messageN}
]
}

the JSONMap Parser, would have a new configuration item for message
selection, that would be a JSON Path expression

“messageSelector” : “$.messages “

Inside the JSONMap Parser, it would evaluate the expression, and do the
same processing on each item returned by the expression list.

the Parser interface already supports returning multiple message objects
from a single byte[] input.

There is a performance penalty to be paid here, and it is more than just
doing more than one message due to the JSONPath evaluation.

I can see this being useful in a couple of circumstances:

   -

   You want to work with some document format with metron but do not have
   NiFi or the equivalent available or setup yet
   -

   You want to prototype with Metron before you get the ‘preprocessing’
   setup
   -

   You are not going to be able to use NiFi and are ok with the performance

I have something in github to look at for more detail :
ottobackwards/json-path-play


Thoughts?

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/903
  
Failure during vagrant up for metron-on-ubuntu

```

2018-01-25 10:41:49,302 p=37541 u=ottofowler |  fatal: [node1]: 
FAILED! => {"changed": true, "cmd": "dpkg-scanpackages . /dev/null | gzip -9c > 
dists/METRON/main/binary-amd64/Packages.gz", "delta": "0:00:00.062622", "end": 
"2018-01-25 15:41:49.325958", "failed": true, "rc": 2, "start": "2018-01-25 
15:41:49.263336", "stderr": "/bin/sh: 1: cannot create 
dists/METRON/main/binary-amd64/Packages.gz: Directory 
nonexistent\ndpkg-scanpackages: info: Wrote 0 entries to output Packages 
file.", "stdout": "", "stdout_lines": [], "warnings": []}
2018-01-25 10:41:49,332 p=37541 u=ottofowler |    
< PLAY RECAP >
  
\   ^__^
 \  (oo)\___
(__)\   )\/\
||w |
|| ||
```


--
platform_info
--

```
Metron 0.4.3
--
* pr-903
--
commit d2cf8fca52d746ec65597cbb826f4cb9ac886a21
Author: Nick Allen 
Date:   Wed Jan 24 19:07:49 2018 -0500

The sensors and pcap replay need to start themselves since Monit will 
not do it
--
--
ansible 2.0.0.2
  config file = 
/Users/ottofowler/tmp/metron-pr-903/metron-deployment/vagrant/metron-on-ubuntu/ansible.cfg
  configured module search path = ../../ansible/extra_modules
--
Vagrant 2.0.1
--
Python 2.7.14
--
Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 
2017-10-18T03:58:13-04:00)
Maven home: /usr/local/Cellar/maven/3.5.2/libexec
Java version: 1.8.0_121, vendor: Oracle Corporation
Java home: 
/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.13.3", arch: "x86_64", family: "mac"
--
Docker version 17.12.0-ce, build c97c6d6
--
node
v6.10.2
--
npm
3.10.10
--
Apple LLVM version 9.0.0 (clang-900.0.39.2)
Target: x86_64-apple-darwin17.4.0
Thread model: posix
InstalledDir: 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
--
Compiler is C++11 compliant
--
Darwin Winterfell 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 
PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64
--
Total System Memory = 16384 MB
Processor Model: Intel(R) Core(TM) i7-4870HQ CPU 
Processor Speed: 2.50GHz
Total Physical Processors: 4
Total cores: 4
Disk information:
/dev/disk1s1   465Gi  380Gi   81Gi83% 3929490 92233720368508463170% 
  /
/dev/disk1s4   465Gi  3.0Gi   81Gi 4%   3 92233720368547758040% 
  /private/var/vm
This CPU appears to support virtualization
```


---

[DISCUSS] Using JSON Path to support more complex documents with the JSONMap Parser

[Otto Fowler]

While it would be preferred if all data streamed into the parsers is
already in ‘stream’ form, as opposed to ‘batched’ form, it may not always
be possible, or possible at every step of system development.

I was wondering if it would be worth adding optional support to the JSONMap
Parser to support more complex documents, and split them in the parser into
multiple messages. This is similar in function to the JSON Splitter
processor in NiFi

So, a document would come into the JSONMap Parser from Kafka, with some
embedded set of the real message content, such as in this simplified
example:

{
“messages" : [
{ message1},
{ message2},
….
{messageN}
]
}

the JSONMap Parser, would have a new configuration item for message
selection, that would be a JSON Path expression

“messageSelector” : “$.messages “

Inside the JSONMap Parser, it would evaluate the expression, and do the
same processing on each item returned by the expression list.

the Parser interface already supports returning multiple message objects
from a single byte[] input.

There is a performance penalty to be paid here, and it is more than just
doing more than one message due to the JSONPath evaluation.

I can see this being useful in a couple of circumstances:

   -

   You want to work with some document format with metron but do not have
   NiFi or the equivalent available or setup yet
   -

   You want to prototype with Metron before you get the ‘preprocessing’
   setup
   -

   You are not going to be able to use NiFi and are ok with the performance

I have something in github to look at for more detail :
ottobackwards/json-path-play


Thoughts?

[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/903
  
Trying this now.   
Only comment on the content here is there is a _lot_ going on in this pr.  
A lot of while I'm here I might as well work.
It might have been better to have kept this more narrow ( although it was 
never going to be narrow ).
I wouldn't change anything at this point however.


---

[GitHub] metron pull request #901: METRON-1410 Check for existing HBASE tables before...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/901#discussion_r163868868
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
 ---
@@ -190,7 +190,7 @@ def create_hbase_tables(self):
   self.__params.hbase_principal_name,
   execute_user=self.__params.hbase_user)
 
-cmd = "echo \"create '{0}','{1}'\" | hbase shell -n"
+cmd = "if [[ $(echo \"exists '{0}'\" | hbase shell | grep 'not 
exist') ]]; then echo \"create '{0}','{1}'\" | hbase shell -n; fi"
--- End diff --

done



---

[GitHub] metron pull request #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[cestella]

GitHub user cestella reopened a pull request:

https://github.com/apache/metron/pull/907

METRON-1427: Add support for storm 1.1 and hdp 2.6

## Contributor Comments
Right now our ambari mpack won't run cleanly on HDP 2.6 and Storm 1.1 
because of some classpath issues and not supporting the stack.  We should 
migrate fulldev to run 2.6 (while still working with 2.5) and validate that we 
work with storm 1.1.   The only code changes in here is relocating and shading 
jackson (kerberized storm 1.1 requires a conflicting version of jackson from ES 
and us, so we shade and relocate it in parsers, but didn't in profiler and 
elasticsearch uber jars.  This PR corrects that).  


You can validate this by:
* Running up full-dev and ensuring data flows through
* Kerberizing full-dev and ensure things still work

I have done the above, but you can feel free to do the same.  
## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron 2.6_investigation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/907.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #907


commit 4486b7ff58bfb98a5bc1ed8e05490fb7202bab9f
Author: cstella 
Date:   2018-01-22T22:11:18Z

Updating the stack definitions and such on the mpack to 2.6

commit 972fe01e19e28f3be94393d83b49a269dbc09e9c
Author: cstella 
Date:   2018-01-23T17:10:38Z

random service check bug

commit 5d56bc1fba811630554192e2fc8614b2050d520d
Author: cstella 
Date:   2018-01-24T00:36:37Z

Updating poms to relocate jackson.

commit b89fa88ddd769aece0e211c4b87a4b0bb15aa309
Author: cstella 
Date:   2018-01-24T16:32:05Z

Added comments.

commit 8e03c293d01ce33026c87302a821fa9a5b91c8dc
Author: cstella 
Date:   2018-01-24T19:54:02Z

Merge branch 'master' into 2.6_investigation

commit c13956ed594bcdaf26923fa49f15690949850fb5
Author: cstella 
Date:   2018-01-25T14:17:12Z

Merge branch 'master' into 2.6_investigation

commit 3969a67a24cf07825fe2cd0a2418c9654b7e78c8
Author: cstella 
Date:   2018-01-25T14:28:43Z

Normalizing security protocols to the one that kafka supports.




---

[GitHub] metron pull request #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[cestella]

Github user cestella closed the pull request at:

https://github.com/apache/metron/pull/907


---

[GitHub] metron issue #909: METRON-1429: SearchIntegrationTest refactor

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/909
  
@cestella You're right about that - the general goodness. We (I) had made 
the mistake of doing some test refactoring in the ES upgrade branch that would 
have been better done in a separate PR, committed to master, and then pulled 
into the ES branch. Splitting out improvements/refactorings the way Ryan did 
here means everyone gets the benefit, and further changes or enhancements don't 
get mangled in repeated merges in the Solr feature branch.


---

[GitHub] metron issue #909: METRON-1429: SearchIntegrationTest refactor

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/909
  
+1 pending travis


---

[GitHub] metron issue #909: METRON-1429: SearchIntegrationTest refactor

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/909
  
Sounds good, +1  Good work here, that test was confusing.


---

[GitHub] metron issue #909: METRON-1429: SearchIntegrationTest refactor

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/909
  
I think this is useful outside of any Solr work and I intended for it to go 
into master.


---

[GitHub] metron issue #909: METRON-1429: SearchIntegrationTest refactor

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/909
  
@cestella This is going into master per the merge notes, right?


---

[GitHub] metron issue #909: METRON-1429: SearchIntegrationTest refactor

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/909
  
Should this be against master or should this be committed against the Solr 
branch?  It *seems* like this is general purpose goodness and maybe fits in 
master, but I wanted to double check.


---

[GitHub] metron pull request #909: METRON-1429: SearchIntegrationTest refactor

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/909#discussion_r163859481
  
--- Diff: 
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java
 ---
@@ -724,6 +503,52 @@ public void sort_query_sorts_results_ascending() 
throws Exception {
 }
   }
 
+  @Test
+  public void sort_ascending_with_missing_fields() throws Exception {
+SearchRequest request = 
JSONUtils.INSTANCE.load(sortAscendingWithMissingFields, SearchRequest.class);
+SearchResponse response = dao.search(request);
+Assert.assertEquals(10, response.getTotal());
+List results = response.getResults();
+Assert.assertEquals(10, results.size());
+
+// the remaining are missing the 'threat:triage:score' and should be 
sorted last
+
Assert.assertFalse(results.get(0).getSource().containsKey("threat:triage:score"));
+
Assert.assertFalse(results.get(1).getSource().containsKey("threat:triage:score"));
+
Assert.assertFalse(results.get(2).getSource().containsKey("threat:triage:score"));
+
Assert.assertFalse(results.get(3).getSource().containsKey("threat:triage:score"));
+
Assert.assertFalse(results.get(4).getSource().containsKey("threat:triage:score"));
+
Assert.assertFalse(results.get(5).getSource().containsKey("threat:triage:score"));
+
Assert.assertFalse(results.get(6).getSource().containsKey("threat:triage:score"));
+
Assert.assertFalse(results.get(7).getSource().containsKey("threat:triage:score"));
+
+// validate sorted order - there are only 2 with a 
'threat:triage:score'
+Assert.assertEquals("10", 
results.get(8).getSource().get("threat:triage:score"));
+Assert.assertEquals("20", 
results.get(9).getSource().get("threat:triage:score"));
+  }
+
+  @Test
+  public void sort_descending_with_missing_fields() throws Exception {
+SearchRequest request = 
JSONUtils.INSTANCE.load(sortDescendingWithMissingFields, SearchRequest.class);
+SearchResponse response = dao.search(request);
+Assert.assertEquals(10, response.getTotal());
+List results = response.getResults();
+Assert.assertEquals(10, results.size());
+
+// validate sorted order - there are only 2 with a 
'threat:triage:score'
+Assert.assertEquals("20", 
results.get(0).getSource().get("threat:triage:score"));
+Assert.assertEquals("10", 
results.get(1).getSource().get("threat:triage:score"));
+
+// the remaining are missing the 'threat:triage:score' and should be 
sorted last
+
Assert.assertFalse(results.get(2).getSource().containsKey("threat:triage:score"));
--- End diff --

Done


---

[GitHub] metron pull request #909: METRON-1429: SearchIntegrationTest refactor

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/909#discussion_r163859462
  
--- Diff: 
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java
 ---
@@ -724,6 +503,52 @@ public void sort_query_sorts_results_ascending() 
throws Exception {
 }
   }
 
+  @Test
+  public void sort_ascending_with_missing_fields() throws Exception {
+SearchRequest request = 
JSONUtils.INSTANCE.load(sortAscendingWithMissingFields, SearchRequest.class);
+SearchResponse response = dao.search(request);
+Assert.assertEquals(10, response.getTotal());
+List results = response.getResults();
+Assert.assertEquals(10, results.size());
+
+// the remaining are missing the 'threat:triage:score' and should be 
sorted last
+
Assert.assertFalse(results.get(0).getSource().containsKey("threat:triage:score"));
--- End diff --

Done


---

[GitHub] metron pull request #906: METRON-1426: SensorIndexingConfigControllerIntegra...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/906


---

[GitHub] metron issue #906: METRON-1426: SensorIndexingConfigControllerIntegrationTes...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/906
  
+1


---

[GitHub] metron issue #902: METRON-1413 Add Metron Commit Tool

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/902
  
I have another script I want to add for tracking master in feature branches 
after this as well


---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/907
  
Hi @cestella , I did a 12-node deploy on CentOS 7 with this patch. 
Post-kerberization, I noticed the following errors in Metron REST. Is this a 
related issue or a different one?

```
18/01/25 10:16:02 ERROR boot.SpringApplication: Application startup failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
creating bean with name 'alertController': Unsatisfied dependency expressed 
through field 'alertService'; nested exception is 
org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
creating bean with name 'alertServiceImpl' defined in URL 
[jar:file:/usr/metron/0.4.3/lib/metron-rest-0.4.3.jar!/org/apache/metron/rest/service/impl/AlertServiceImpl.class]:
 Unsatisfied dependency expressed through constructor parameter 0; nested 
exception is org.springframework.beans.factory.UnsatisfiedDependencyException: 
Error creating bean with name 'kafkaServiceImpl' defined in URL 
[jar:file:/usr/metron/0.4.3/lib/metron-rest-0.4.3.jar!/org/apache/metron/rest/service/impl/KafkaServiceImpl.class]:
 Unsatisfied dependency expressed through constructor parameter 2; nested 
exception is org.springframework.beans.factory.BeanCreationException: Error 
creating bean with name 'kafkaProducer' defined in class path resourc
 e [org/apache/metron/rest/config/KafkaConfig.class]: Bean instantiation via 
factory method failed; nested exception is 
org.springframework.beans.BeanInstantiationException: Failed to instantiate 
[org.apache.kafka.clients.producer.KafkaProducer]: Factory method 
'kafkaProducer' threw exception; nested exception is 
org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at 
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:569)
at 
org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at 
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:349)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1219)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at 
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at 
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at 
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at 
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at 
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751)
at 
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:861)
at 
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541)
at 
org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
at 
org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761)
at 
org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:371)
at 
org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at 
org.springframework.boot.SpringApplication.run(SpringApplication.java:1186)
at 
org.springframework.boot.SpringApplication.run(SpringApplication.java:1175)
at 
org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:29)
Caused by: 
org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
creating bean with name 'alertServiceImpl' defined in URL 
[jar:file:/usr/metron/0.4.3/lib/metron-rest-0.4.3.jar!/org/apache/metron/rest/service/impl/AlertServiceImpl.class]:
 Unsatisfied dependency expressed through constructor parameter 0; nested 
exception is org.springframework.beans.factory.UnsatisfiedDependencyException: 
Error creating bean with name 'kafkaServiceImpl' defined in URL 
[jar:file:/usr/metron/0.4.3/lib/metron-rest-0.4.3.jar!/org/apache/metron/rest/service/impl/KafkaServiceImpl.class]:
 Unsatisfied dependency expressed through constructor parameter 2; nested 
exception is org.springframework.beans.factory.BeanCreationException: Error 
creating bean with