Re: Kafka error when upgrading to Metron 0.3.0

2017-01-08 Thread Michael Miklavcic 
Hi Tyler, I don't recall seeing any failures the last time I ran this. I
will take a look.

On Jan 8, 2017 9:56 PM, "Tyler Moore"  wrote:

Michael,

I am receiving error when trying to build with HPD profile, have you had
any problems with pcap-backend tests failing when building with hdp profile?
Errors and log files provided below:

Results :

Failed tests:
  
PcapTopologyIntegrationTest.testTimestampInKey:152->testTopology:388->assertInOrder:542
null
  
PcapTopologyIntegrationTest.testTimestampInPacket:135->testTopology:388->assertInOrder:542
null



Tests run: 2, Failures: 2, Errors: 0, Skipped: 0

[INFO] 

[INFO] Reactor Summary:
[INFO]
[INFO] Metron . SUCCESS [
 0.639 s]
[INFO] metron-analytics ... SUCCESS [
 0.029 s]
[INFO] metron-maas-common . SUCCESS [
 8.771 s]
[INFO] metron-platform  SUCCESS [
 0.057 s]
[INFO] metron-test-utilities .. SUCCESS [
 0.976 s]
[INFO] metron-integration-test  SUCCESS [
 3.888 s]
[INFO] metron-maas-service  SUCCESS [01:19
min]
[INFO] metron-common .. SUCCESS [
41.718 s]
[INFO] metron-statistics .. SUCCESS [
30.713 s]
[INFO] metron-hbase ... SUCCESS [
37.286 s]
[INFO] metron-profiler-common . SUCCESS [
 7.967 s]
[INFO] metron-profiler-client . SUCCESS [
54.088 s]
[INFO] metron-profiler  SUCCESS [02:28
min]
[INFO] metron-writer .. SUCCESS [
 7.151 s]
[INFO] metron-enrichment .. SUCCESS [
57.076 s]
[INFO] metron-indexing  SUCCESS [
 7.028 s]
[INFO] metron-solr  SUCCESS [
36.695 s]
[INFO] metron-pcap  SUCCESS [
 1.123 s]
[INFO] metron-parsers . SUCCESS [01:58
min]
[INFO] metron-pcap-backend  FAILURE [
49.928 s]
[INFO] metron-data-management . SKIPPED
[INFO] metron-api . SKIPPED
[INFO] metron-management .. SKIPPED
[INFO] elasticsearch-shaded ... SKIPPED
[INFO] metron-elasticsearch ... SKIPPED
[INFO] metron-deployment .. SKIPPED
[INFO] Metron Ambari Management Pack .. SKIPPED
[INFO] 

[INFO] BUILD FAILURE
[INFO] 

[INFO] Total time: 11:31 min
[INFO] Finished at: 2017-01-08T22:23:59-05:00
[INFO] Final Memory: 287M/6202M
[INFO] 

[ERROR] Failed to execute goal org.apache.maven.plugins:
maven-surefire-plugin:2.18:test (integration-tests) on project
metron-pcap-backend: There are test failures.


Regards,

Tyler Moore
Software Engineer
Phone: 248-909-2769 <(248)%20909-2769>
Email: moore.ty...@goflyball.com


On Thu, Jan 5, 2017 at 5:21 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Hey Tyler,
>
> Build Metron with the HDP profile to get the proper deps for this
> -PHDP-2.5.0.0
>
> Hopefully that works for you.
>
> Best,
> Mike
>
>
> On Thu, Jan 5, 2017 at 3:17 PM, Tyler Moore  wrote:
>
> > Hey all,
> >
> > Wondering if there is a solution to the "Offset lags for kafka not
> > supported for older versions. Please update kafka spout to latest
> version."
> > error seen when upgrading to metron 0.3.0?
> >
> > I know it is due to kafka-storm dependency that needs updated, but what
> is
> > the best way to go about that? Is it as simple at changing the version in
> > the metron pom.xml and provisioning?
> >
> > Regards,
> >
> > Tyler Moore
> > Software Engineer
> > Phone: 248-909-2769
> > Email: moore.ty...@goflyball.com
> >
>

Re: Kafka error when upgrading to Metron 0.3.0

2017-01-08 Thread Tyler Moore 
Michael,

I am receiving error when trying to build with HPD profile, have you had
any problems with pcap-backend tests failing when building with hdp profile?
Errors and log files provided below:

Results :

Failed tests:

PcapTopologyIntegrationTest.testTimestampInKey:152->testTopology:388->assertInOrder:542
null

PcapTopologyIntegrationTest.testTimestampInPacket:135->testTopology:388->assertInOrder:542
null



Tests run: 2, Failures: 2, Errors: 0, Skipped: 0

[INFO]

[INFO] Reactor Summary:
[INFO]
[INFO] Metron . SUCCESS [
 0.639 s]
[INFO] metron-analytics ... SUCCESS [
 0.029 s]
[INFO] metron-maas-common . SUCCESS [
 8.771 s]
[INFO] metron-platform  SUCCESS [
 0.057 s]
[INFO] metron-test-utilities .. SUCCESS [
 0.976 s]
[INFO] metron-integration-test  SUCCESS [
 3.888 s]
[INFO] metron-maas-service  SUCCESS [01:19
min]
[INFO] metron-common .. SUCCESS [
41.718 s]
[INFO] metron-statistics .. SUCCESS [
30.713 s]
[INFO] metron-hbase ... SUCCESS [
37.286 s]
[INFO] metron-profiler-common . SUCCESS [
 7.967 s]
[INFO] metron-profiler-client . SUCCESS [
54.088 s]
[INFO] metron-profiler  SUCCESS [02:28
min]
[INFO] metron-writer .. SUCCESS [
 7.151 s]
[INFO] metron-enrichment .. SUCCESS [
57.076 s]
[INFO] metron-indexing  SUCCESS [
 7.028 s]
[INFO] metron-solr  SUCCESS [
36.695 s]
[INFO] metron-pcap  SUCCESS [
 1.123 s]
[INFO] metron-parsers . SUCCESS [01:58
min]
[INFO] metron-pcap-backend  FAILURE [
49.928 s]
[INFO] metron-data-management . SKIPPED
[INFO] metron-api . SKIPPED
[INFO] metron-management .. SKIPPED
[INFO] elasticsearch-shaded ... SKIPPED
[INFO] metron-elasticsearch ... SKIPPED
[INFO] metron-deployment .. SKIPPED
[INFO] Metron Ambari Management Pack .. SKIPPED
[INFO]

[INFO] BUILD FAILURE
[INFO]

[INFO] Total time: 11:31 min
[INFO] Finished at: 2017-01-08T22:23:59-05:00
[INFO] Final Memory: 287M/6202M
[INFO]

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.18:test
(integration-tests) on project metron-pcap-backend: There are test failures.


Regards,

Tyler Moore
Software Engineer
Phone: 248-909-2769
Email: moore.ty...@goflyball.com


On Thu, Jan 5, 2017 at 5:21 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Hey Tyler,
>
> Build Metron with the HDP profile to get the proper deps for this
> -PHDP-2.5.0.0
>
> Hopefully that works for you.
>
> Best,
> Mike
>
>
> On Thu, Jan 5, 2017 at 3:17 PM, Tyler Moore  wrote:
>
> > Hey all,
> >
> > Wondering if there is a solution to the "Offset lags for kafka not
> > supported for older versions. Please update kafka spout to latest
> version."
> > error seen when upgrading to metron 0.3.0?
> >
> > I know it is due to kafka-storm dependency that needs updated, but what
> is
> > the best way to go about that? Is it as simple at changing the version in
> > the metron pom.xml and provisioning?
> >
> > Regards,
> >
> > Tyler Moore
> > Software Engineer
> > Phone: 248-909-2769
> > Email: moore.ty...@goflyball.com
> >
>
---
Test set: org.apache.metron.pcap.integration.PcapTopologyIntegrationTest
---
Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 33.401 sec <<< 
FAILURE! - in org.apache.metron.pcap.integration.PcapTopologyIntegrationTest
testTimestampInPacket(org.apache.metron.pcap.integration.PcapTopologyIntegrationTest)
  Time elapsed: 19.976 sec  <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at 
org.apache.metron.pcap.integration.PcapTopologyIntegrationTest.assertInOrder(PcapTopologyIntegrationTest.java:542)
at

Re: Enrich enrichment

2017-01-08 Thread Casey Stella 
You could do the geo enrichment normally and do a stellar hbase enrichment
in the threat Intel phase.

On Sun, Jan 8, 2017 at 16:22 Ryan Merriman  wrote:

> Hbase enrichments and geo enrichments are done in parallel so I would not
> expect this to work.  You could do the Hbase enrichment as a threat Intel
> enrichment and that should work because enrichments and threat Intel are
> done in series.
>
>
>
> The ideal way would be to chain together Stellar enrichments but I don't
> think there is a geo enrichment function created yet.  I think that should
> be a Jira.  I know someone is working on an update to how we do geo
> enrichments so I will file a follow on Jira if it's not included in the
> scope of that work.
>
>
>
> Ryan
>
>
>
> > On Jan 8, 2017, at 2:31 PM, Dima Kovalyov 
> wrote:
>
> >
>
> > Is it possible to enrich enrichment?
>
> >
>
> > For example I have IP address, I enrich it with geo and get City name,
>
> > now I want to enrich City name with city crime level (assume I have that
>
> > data). But when I do that it just does not work. I specify enrichment
>
> > like that:
>
> >> {
>
> >>  "index" : "msexchange",
>
> >>  "batchSize" : 5,
>
> >>  "enrichment" : {
>
> >>"fieldMap" : {
>
> >>  "geo" : [ "destination_ip", "source_ip" ],
>
> >>  "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ],
>
> >>"hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ],
>
> >>"hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ]
>
> >>},
>
> >>"fieldToTypeMap" : {
>
> >>  "enrichments.geo.destination_ip.country" : [ "city_crime_level" ],
>
> >>  "enrichments:geo:destination_ip:country" : [ "city_crime_level" ],
>
> >>  "enrichments.geo.destination_ip:country" : [ "city_crime_level" ]
>
> >>},
>
> >>"config" : { }
>
> >>  },
>
> >>  "threatIntel" : {
>
> >>"fieldMap" : { },
>
> >>"fieldToTypeMap" : { },
>
> >>"config" : { },
>
> >>"triageConfig" : {
>
> >>  "riskLevelRules" : { },
>
> >>  "aggregator" : "MAX",
>
> >>  "aggregationConfig" : { }
>
> >>}
>
> >>  },
>
> >>  "configuration" : { }
>
> >> }
>
> > I tried all the ways how enrichment field can be entered just to be sure
>
> > I do not mistype it.
>
> >
>
> > - Dima
>
>

Re: Enrich enrichment

2017-01-08 Thread Ryan Merriman 
Hbase enrichments and geo enrichments are done in parallel so I would not 
expect this to work.  You could do the Hbase enrichment as a threat Intel 
enrichment and that should work because enrichments and threat Intel are done 
in series.

The ideal way would be to chain together Stellar enrichments but I don't think 
there is a geo enrichment function created yet.  I think that should be a Jira. 
 I know someone is working on an update to how we do geo enrichments so I will 
file a follow on Jira if it's not included in the scope of that work.

Ryan

> On Jan 8, 2017, at 2:31 PM, Dima Kovalyov  wrote:
> 
> Is it possible to enrich enrichment?
> 
> For example I have IP address, I enrich it with geo and get City name,
> now I want to enrich City name with city crime level (assume I have that
> data). But when I do that it just does not work. I specify enrichment
> like that:
>> {
>>  "index" : "msexchange",
>>  "batchSize" : 5,
>>  "enrichment" : {
>>"fieldMap" : {
>>  "geo" : [ "destination_ip", "source_ip" ],
>>  "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ],
>>"hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ],
>>"hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ]
>>},
>>"fieldToTypeMap" : {
>>  "enrichments.geo.destination_ip.country" : [ "city_crime_level" ],
>>  "enrichments:geo:destination_ip:country" : [ "city_crime_level" ],
>>  "enrichments.geo.destination_ip:country" : [ "city_crime_level" ]
>>},
>>"config" : { }
>>  },
>>  "threatIntel" : {
>>"fieldMap" : { },
>>"fieldToTypeMap" : { },
>>"config" : { },
>>"triageConfig" : {
>>  "riskLevelRules" : { },
>>  "aggregator" : "MAX",
>>  "aggregationConfig" : { }
>>}
>>  },
>>  "configuration" : { }
>> }
> I tried all the ways how enrichment field can be entered just to be sure
> I do not mistype it.
> 
> - Dima

Enrich enrichment

2017-01-08 Thread Dima Kovalyov 
Is it possible to enrich enrichment?

For example I have IP address, I enrich it with geo and get City name,
now I want to enrich City name with city crime level (assume I have that
data). But when I do that it just does not work. I specify enrichment
like that:
> {
>   "index" : "msexchange",
>   "batchSize" : 5,
>   "enrichment" : {
> "fieldMap" : {
>   "geo" : [ "destination_ip", "source_ip" ],
>   "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ],
> "hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ],
> "hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ]
> },
> "fieldToTypeMap" : {
>   "enrichments.geo.destination_ip.country" : [ "city_crime_level" ],
>   "enrichments:geo:destination_ip:country" : [ "city_crime_level" ],
>   "enrichments.geo.destination_ip:country" : [ "city_crime_level" ]
> },
> "config" : { }
>   },
>   "threatIntel" : {
> "fieldMap" : { },
> "fieldToTypeMap" : { },
> "config" : { },
> "triageConfig" : {
>   "riskLevelRules" : { },
>   "aggregator" : "MAX",
>   "aggregationConfig" : { }
> }
>   },
>   "configuration" : { }
> }
I tried all the ways how enrichment field can be entered just to be sure
I do not mistype it.

- Dima