Re: Removing and renaming fields with STELLAR

[James Sirota]

Do you still need help? Or were you able to do what you wanted? 

27.09.2016, 12:34, "Otto Fowler" :
> I’ll get there
>
> --
>
> On September 27, 2016 at 15:23:10, Casey Stella (ceste...@gmail.com) wrote:
>
> Yeah, it's confusing, so I feel your pain. I made a "Navigating the
> Architecture" (which should probably be named "Navigating the Developer
> Docs" ;) section in the top-level doc here
> 
> that
> links to the docs for various parts of the architecture.
>
> Hope that's helpful in the future! :)
>
> On Tue, Sep 27, 2016 at 3:05 PM, Otto Fowler 
> wrote:
>
>>  Ok, I was going by the readme in metron-common and the wiki.
>>  I’ll look in both readme files next time
>>
>>  --
>>
>>  Sent with Airmail
>>
>>  On September 27, 2016 at 14:49:29, Casey Stella (ceste...@gmail.com)
>>  wrote:
>>
>>  It's described in https://github.com/apache/incubator-metron/tree/master/
>>  metron-platform/metron-parsers#fieldtransformation-configuration under
>>  "REMOVE". The wiki tends to lag; the README.md docs are mostly
>
> up-to-date.
>>  Casey
>>
>>  On Tue, Sep 27, 2016 at 2:47 PM, Otto Fowler 
>>  wrote:
>>
>>  > I’m going by the wiki and REMOVE is not in there. Where should I look
>
> for
>>  > the most up to date stuff? The unit tests?
>>  >
>>  >
>>  >
>>  > On September 27, 2016 at 14:33:08, Casey Stella (ceste...@gmail.com)
>>  > wrote:
>>  >
>>  > Yes, it is in the parser. You can assign a field to NULL and it will
>>  > remove it.
>>  >
>>  > Consider the following example parser config which takes the foo field
>>  and
>>  > renames it to bar:
>>  >
>>  > {
>>  > "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
>>  > "sensorTopic":"custom",
>>  > "parserConfig" : {
>>  > "mapStrategy" : "UNFOLD"
>>  > },"fieldTransformations" : [
>>  > {
>>  > "transformation" : "STELLAR"
>>  > ,"output" : [ "bar", "foo" ]
>>  > ,"config" : {
>>  > "bar" : "foo"
>>  > ,"foo" : "NULL"
>>  > }
>>  > }
>>  > ]
>>  > }
>>  >
>>  > Note the output section, that defines the order in which the stellar
>>  > statements are executed. You first set bar to whatever is in foo. Then
>>  > you set foo to NULL, which will delete it. You will be left with just
>>  bar,
>>  > which will contain what is in foo.
>>  >
>>  > Another way to do this is to use the REMOVE transformation
>>  > <
>>  > https://github.com/apache/incubator-metron/tree/master/
>>  > metron-platform/metron-parsers#fieldtransformation-configuration>
>>  >
>>  > in
>>  > combination with the stellar transformation:
>>  > {
>>  > "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
>>  > "sensorTopic":"custom",
>>  > "parserConfig" : {
>>  > "mapStrategy" : "UNFOLD"
>>  > },"fieldTransformations" : [
>>  > {
>>  > "transformation" : "STELLAR"
>>  > ,"output" : [ "bar", "foo" ]
>>  > ,"config" : {
>>  > "bar" : "foo"
>>  > }
>>  > },
>>  > {
>>  > "input" : "foo"
>>  > , "transformation" : "REMOVE"
>>  > }
>>  > ]
>>  > }
>>  >
>>  > Best,
>>  >
>>  > Casey
>>  >
>>  > On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler 
>>  > wrote:
>>  >
>>  > > Is it possible to remove and rename fields with stellar?
>>  > > I have “foreign” json coming out of the JSONParser, and I need to
>>  rename
>>  > > some fields to the core metron json object names and possibly remove
>>  some
>>  > > other fields.
>>  > >
>>  >

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

[DISCUSS] Elastic Search Tuning

[James Sirota]

I wanted to start a thread where people can post their Elastic Search tuning 
tips for the Metron ingest use case.  ES is commonly the bottleneck in the 
ingest pipeline and it would be great if people could share tips for how to 
tune it. 

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Re: Metron with HDP 2.5

[Dima Kovalyov]

Thank you for reply James,

I have performed Ambari 2.2 -> 2.4 and HDP 2.4 -> 2.5 upgrade on
quick-dev-platform in Vagrant Metron setup. Most of it went just fine,
except few intermittent errors.

Can you please advise where do you want it to be documented and/or
discussed?

Right now I am battling issue with Storm. After the upgrade I need to
re-build Storm Topology jars in order to propagate HDP 2.5 files in
them. Can you please advise how I can do that on already running setup?
(TLDR: how to build java jars manually?)

Thank you!

- Dima

On 09/27/2016 06:59 PM, James Sirota wrote:
> We didn't test it enough to know how well it works.  We don't expect it to 
> work at this time. If you can run it up for us and document all the issues 
> that go wrong that would be greatly appreciated. 
>
> 26.09.2016, 16:54, "Satish Abburi" :
>> All, any one using/tested this combination.
>>
>> Thanks,
>> Satish
> --- 
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>

[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser

[kylerichardson]

Github user kylerichardson commented on the issue:

https://github.com/apache/incubator-metron/pull/276
  
Currently my branch doesn't have build_utils. Going to rebase and see if 
that fixes the CI build.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #281: METRON-456: Create stellar management fu...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/incubator-metron/pull/281

METRON-456: Create stellar management functions to test grok statements

Create two stellar management functions:
* GROK_APPLY - Test grok statements
* GROK_PREDICT - Given a string, predict a grok statement that matches it.

A section in the `metron-management` README.md has been added to show these 
functions' as well as the functions provided by #277 to debug and create grok 
statements.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron METRON-456

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/281.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #281


commit 27d7925b74f412e4b8d31a6bf177684d62c0630f
Author: cstella 
Date:   2016-09-27T22:50:58Z

METRON-456: Create stellar management functions to test grok statements

commit 1d946b200a53bbdf2202da3a8ba38e653938711e
Author: cstella 
Date:   2016-09-28T01:34:02Z

Updated docs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser

[kylerichardson]

Github user kylerichardson commented on the issue:

https://github.com/apache/incubator-metron/pull/276
  
I've tested in a slimmed down single node vm (no sensors) but not in 
vagrant.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #280: Metron 398: Bump release version to 0.2.1BETA i...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/280
  
+1 by inspection


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #268: METRON-421 Make Stellar Profiler Client API Acc...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/268
  
Before this the problem was that the `HBaseProfilerClientTest` in 
`metron-profiler-client` could not start the HBase mini-cluster. Seems with the 
latest changes, the problem has migrated to the enrichment integration tests.   
 I'll see what dependencies it is deciding to pull in now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #280: Metron 398: Bump release version to 0.2....

[mmiklavc]

GitHub user mmiklavc opened a pull request:

https://github.com/apache/incubator-metron/pull/280

Metron 398: Bump release version to 0.2.1BETA in master

Completes https://issues.apache.org/jira/browse/METRON-398

Tested this on quick-dev

I first cleared my .m2 cache just to be sure all dependencies are resolved 
using the newest version *only*.
`~/.m2$ rm -rf ./org/apache/metron`

**Testing done**

Full dev build
`incubator-metron$ mvn clean install`
rpm build
`incubator-metron/metron-deployment$ mvn clean install -Pbuild-rpms`
quick-dev build
`incubator-metron/metron-deployment/vagrant/quick-dev-platform$ ./run.sh`


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mmiklavc/incubator-metron METRON-398

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/280.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #280


commit 50202ebe39ef594ffa78e28f57ecf1349e2334a6
Author: Michael Miklavcic 
Date:   2016-08-29T15:35:54Z

METRON-398 Update Metron version from 0.2.0BETA to 0.2.1BETA. Covers metron 
main, metron-analytics, and metron-deployment. Removed old metron-ui references 
in the rat check.

commit fee125fe25086d40018b6128c0cff6225eaaa0c3
Author: Michael Miklavcic 
Date:   2016-09-27T20:34:01Z

METRON-398 Update Metron version from 0.2.0BETA to 0.2.1BETA




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #235: METRON-398 Bump release version to 0.2.1...

[mmiklavc]

Github user mmiklavc closed the pull request at:

https://github.com/apache/incubator-metron/pull/235


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #278: Metron 451 SerDeUtils - java.lang.ClassNotFound...

[danieljue]

Github user danieljue commented on the issue:

https://github.com/apache/incubator-metron/pull/278
  
Thanks for the info @cestella !

I get build success on all the modules if I do `mvn clean install -U 
-DskipTests=true`  from the top level Metron directory.  ;)

I think the integrations tests in maas-common that are getting triggered 
are causing builds to fail for me.  (Could be others if it would get passed 
that).

I'll open a new ticket so this doesn't get lost in the weeds on this thread.





---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Removing and renaming fields with STELLAR

[Otto Fowler]

I’ll get there

-- 


On September 27, 2016 at 15:23:10, Casey Stella (ceste...@gmail.com) wrote:

Yeah, it's confusing, so I feel your pain. I made a "Navigating the
Architecture" (which should probably be named "Navigating the Developer
Docs" ;) section in the top-level doc here

that
links to the docs for various parts of the architecture.

Hope that's helpful in the future! :)

On Tue, Sep 27, 2016 at 3:05 PM, Otto Fowler 
wrote:

> Ok, I was going by the readme in metron-common and the wiki.
> I’ll look in both readme files next time
>
> --
>
> Sent with Airmail
>
> On September 27, 2016 at 14:49:29, Casey Stella (ceste...@gmail.com)
> wrote:
>
> It's described in https://github.com/apache/incubator-metron/tree/master/
> metron-platform/metron-parsers#fieldtransformation-configuration under
> "REMOVE". The wiki tends to lag; the README.md docs are mostly
up-to-date.
>
> Casey
>
> On Tue, Sep 27, 2016 at 2:47 PM, Otto Fowler 
> wrote:
>
> > I’m going by the wiki and REMOVE is not in there. Where should I look
for
> > the most up to date stuff? The unit tests?
> >
> >
> >
> > On September 27, 2016 at 14:33:08, Casey Stella (ceste...@gmail.com)
> > wrote:
> >
> > Yes, it is in the parser. You can assign a field to NULL and it will
> > remove it.
> >
> > Consider the following example parser config which takes the foo field
> and
> > renames it to bar:
> >
> > {
> > "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
> > "sensorTopic":"custom",
> > "parserConfig" : {
> > "mapStrategy" : "UNFOLD"
> > },"fieldTransformations" : [
> > {
> > "transformation" : "STELLAR"
> > ,"output" : [ "bar", "foo" ]
> > ,"config" : {
> > "bar" : "foo"
> > ,"foo" : "NULL"
> > }
> > }
> > ]
> > }
> >
> > Note the output section, that defines the order in which the stellar
> > statements are executed. You first set bar to whatever is in foo. Then
> > you set foo to NULL, which will delete it. You will be left with just
> bar,
> > which will contain what is in foo.
> >
> > Another way to do this is to use the REMOVE transformation
> > <
> > https://github.com/apache/incubator-metron/tree/master/
> > metron-platform/metron-parsers#fieldtransformation-configuration>
> >
> > in
> > combination with the stellar transformation:
> > {
> > "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
> > "sensorTopic":"custom",
> > "parserConfig" : {
> > "mapStrategy" : "UNFOLD"
> > },"fieldTransformations" : [
> > {
> > "transformation" : "STELLAR"
> > ,"output" : [ "bar", "foo" ]
> > ,"config" : {
> > "bar" : "foo"
> > }
> > },
> > {
> > "input" : "foo"
> > , "transformation" : "REMOVE"
> > }
> > ]
> > }
> >
> > Best,
> >
> > Casey
> >
> > On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler 
> > wrote:
> >
> > > Is it possible to remove and rename fields with stellar?
> > > I have “foreign” json coming out of the JSONParser, and I need to
> rename
> > > some fields to the core metron json object names and possibly remove
> some
> > > other fields.
> > >
> >
>

[GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser

[danieljue]

Github user danieljue commented on the issue:

https://github.com/apache/incubator-metron/pull/276
  
FYI the PR for METRON-451 is failing at the same place.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #278: Metron 451 SerDeUtils - java.lang.ClassNotFound...

[danieljue]

Github user danieljue commented on the issue:

https://github.com/apache/incubator-metron/pull/278
  
@nickwallen , interesting because the jodatime dependency is already in 
that CSV (it was included in API, but had to add it to the Common module pom)

Seems to  be failing at the same place as METRON-363 Fix Cisco ASA Parser :

`Sep 27, 2016 1:28:55 AM 
com.google.inject.servlet.InternalServletModule$BackwardsCompatibleServletContextProvider
 get
WARNING: You are attempting to use a deprecated API (specifically, 
attempting to @Inject ServletContext inside an eagerly created singleton. While 
we allow this for backwards compatibility, be warned that this MAY have 
unexpected behavior if you have more than one injector (with ServletModule) 
running in the same JVM. Please consult the Guice documentation at 
http://code.google.com/p/google-guice/wiki/Servlets for more information.
Sep 27, 2016 1:28:55 AM 
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider 
as a provider class
Sep 27, 2016 1:28:55 AM 
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering 
org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSWebServices 
as a root resource class
Sep 27, 2016 1:28:55 AM 
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering 
org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices as a root 
resource class
Sep 27, 2016 1:28:55 AM 
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as 
a provider class
Sep 27, 2016 1:28:55 AM 
com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 
AM'
Sep 27, 2016 1:28:55 AM 
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory 
getComponentProvider
INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to 
GuiceManagedComponentProvider with the scope "Singleton"
Sep 27, 2016 1:28:55 AM 
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory 
getComponentProvider
INFO: Binding org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider to 
GuiceManagedComponentProvider with the scope "Singleton"
Sep 27, 2016 1:28:55 AM 
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory 
getComponentProvider
INFO: Binding 
org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.AHSWebServices 
to GuiceManagedComponentProvider with the scope "Singleton"
Sep 27, 2016 1:28:55 AM 
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory 
getComponentProvider
INFO: Binding 
org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices to 
GuiceManagedComponentProvider with the scope "Singleton"
2016-09-27 01:28:59,179 ERROR [Curator-TreeCache-0] curator.ConnectionState 
(ConnectionState.java:checkTimeouts(200)) - Connection timed out for connection 
string (127.0.0.1:48078) and timeout (15000) / elapsed (31009)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = 
ConnectionLoss
at 
org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197)
at 
org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87)
at 
org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
at 
org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:477)
at 
org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:214)
at 
org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:203)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at 
org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:200)
at 
org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:191)
at 
org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:38)
at 
org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.queryForNames(ServiceDiscoveryImpl.java:276)
at 
org.apache.metron.maas.discovery.ServiceDiscoverer.updateState(ServiceDiscoverer.java:129)
at 
org.apache.metron.maas.discovery.ServiceDiscoverer.lambda$new$2(ServiceDiscoverer.java:93)
at 
org.apache.metron.maas.discovery.ServiceDiscoverer$$Lambda$34/648409124.childEvent(Unknown
 Source)
at 
org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:685)
at 
org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:679)
at 

Re: Removing and renaming fields with STELLAR

[Casey Stella]

It's described in https://github.com/apache/incubator-metron/tree/master/
metron-platform/metron-parsers#fieldtransformation-configuration under
"REMOVE".  The wiki tends to lag; the README.md docs are mostly up-to-date.

Casey

On Tue, Sep 27, 2016 at 2:47 PM, Otto Fowler 
wrote:

> I’m going by the wiki and REMOVE is not in there.  Where should I look for
> the most up to date stuff?  The unit tests?
>
>
>
> On September 27, 2016 at 14:33:08, Casey Stella (ceste...@gmail.com)
> wrote:
>
> Yes, it is in the parser. You can assign a field to NULL and it will
> remove it.
>
> Consider the following example parser config which takes the foo field and
> renames it to bar:
>
> {
> "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
> "sensorTopic":"custom",
> "parserConfig" : {
> "mapStrategy" : "UNFOLD"
> },"fieldTransformations" : [
> {
> "transformation" : "STELLAR"
> ,"output" : [ "bar", "foo" ]
> ,"config" : {
> "bar" : "foo"
> ,"foo" : "NULL"
> }
> }
> ]
> }
>
> Note the output section, that defines the order in which the stellar
> statements are executed. You first set bar to whatever is in foo. Then
> you set foo to NULL, which will delete it. You will be left with just bar,
> which will contain what is in foo.
>
> Another way to do this is to use the REMOVE transformation
> <
> https://github.com/apache/incubator-metron/tree/master/
> metron-platform/metron-parsers#fieldtransformation-configuration>
>
> in
> combination with the stellar transformation:
> {
> "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
> "sensorTopic":"custom",
> "parserConfig" : {
> "mapStrategy" : "UNFOLD"
> },"fieldTransformations" : [
> {
> "transformation" : "STELLAR"
> ,"output" : [ "bar", "foo" ]
> ,"config" : {
> "bar" : "foo"
> }
> },
> {
> "input" : "foo"
> , "transformation" : "REMOVE"
> }
> ]
> }
>
> Best,
>
> Casey
>
> On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler 
> wrote:
>
> > Is it possible to remove and rename fields with stellar?
> > I have “foreign” json coming out of the JSONParser, and I need to rename
> > some fields to the core metron json object names and possibly remove some
> > other fields.
> >
>

Re: Removing and renaming fields with STELLAR

[James Sirota]

i think it would be great if we had something like this.  I can see lots of 
people needing this 

27.09.2016, 11:39, "Casey Stella" :
> Just as an aside, I was thinking that it might be nice to have a special
> THIS object in stellar which would return the current message being
> processed, so you could do stuff like MAP_REMOVE('foo', this) to remove a
> field.
>
> On Tue, Sep 27, 2016 at 2:33 PM, Casey Stella  wrote:
>
>>  Yes, it is in the parser. You can assign a field to NULL and it will
>>  remove it.
>>
>>  Consider the following example parser config which takes the foo field and
>>  renames it to bar:
>>
>>  {
>>    "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
>>    "sensorTopic":"custom",
>>    "parserConfig" : {
>>  "mapStrategy" : "UNFOLD"
>>    },"fieldTransformations" : [
>>  {
>>    "transformation" : "STELLAR"
>>  ,"output" : [ "bar", "foo" ]
>>  ,"config" : {
>>    "bar" : "foo"
>>    ,"foo" : "NULL"
>>  }
>>  }
>> ]
>>  }
>>
>>  Note the output section, that defines the order in which the stellar
>>  statements are executed. You first set bar to whatever is in foo. Then
>>  you set foo to NULL, which will delete it. You will be left with just bar,
>>  which will contain what is in foo.
>>
>>  Another way to do this is to use the REMOVE transformation
>>  
>> 
>>  in
>>  combination with the stellar transformation:
>>  {
>>    "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
>>    "sensorTopic":"custom",
>>    "parserConfig" : {
>>  "mapStrategy" : "UNFOLD"
>>    },"fieldTransformations" : [
>>  {
>>    "transformation" : "STELLAR"
>>  ,"output" : [ "bar", "foo" ]
>>  ,"config" : {
>>    "bar" : "foo"
>>  }
>>  },
>>  {
>>    "input" : "foo"
>>   , "transformation" : "REMOVE"
>>  }
>> ]
>>  }
>>
>>  Best,
>>
>>  Casey
>>
>>  On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler 
>>  wrote:
>>
>>>  Is it possible to remove and rename fields with stellar?
>>>  I have “foreign” json coming out of the JSONParser, and I need to rename
>>>  some fields to the core metron json object names and possibly remove some
>>>  other fields.

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Re: Removing and renaming fields with STELLAR

[Casey Stella]

Just as an aside, I was thinking that it might be nice to have a special
THIS object in stellar which would return the current message being
processed, so you could do stuff like MAP_REMOVE('foo', this) to remove a
field.

On Tue, Sep 27, 2016 at 2:33 PM, Casey Stella  wrote:

> Yes, it is in the parser.  You can assign a field to NULL and it will
> remove it.
>
> Consider the following example parser config which takes the foo field and
> renames it to bar:
>
> {
>   "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
>   "sensorTopic":"custom",
>   "parserConfig" : {
> "mapStrategy" : "UNFOLD"
>   },"fieldTransformations" : [
> {
>   "transformation" : "STELLAR"
> ,"output" : [ "bar", "foo" ]
> ,"config" : {
>   "bar" : "foo"
>   ,"foo" : "NULL"
> }
> }
>]
> }
>
> Note the output section, that defines the order in which the stellar
> statements are executed.  You first set bar to whatever is in foo.  Then
> you set foo to NULL, which will delete it.  You will be left with just bar,
> which will contain what is in foo.
>
> Another way to do this is to use the REMOVE transformation
> 
>  in
> combination with the stellar transformation:
> {
>   "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
>   "sensorTopic":"custom",
>   "parserConfig" : {
> "mapStrategy" : "UNFOLD"
>   },"fieldTransformations" : [
> {
>   "transformation" : "STELLAR"
> ,"output" : [ "bar", "foo" ]
> ,"config" : {
>   "bar" : "foo"
> }
> },
> {
>   "input" : "foo"
>  , "transformation" : "REMOVE"
> }
>]
> }
>
> Best,
>
> Casey
>
> On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler 
> wrote:
>
>> Is it possible to remove and rename fields with stellar?
>> I have “foreign” json coming out of the JSONParser, and I need to rename
>> some fields to the core metron json object names and possibly remove some
>> other fields.
>>
>
>

Re: Complete steps to add a new parser

[Otto Fowler]

Thanks Nick,
That is some of the stuff that I have found trying to track down the deploy
bits of the existing parsers, but I don’t want to miss anything, so I’d
like some guidence.  Right now, I’m OK with doing it all inside the metron
ansible base.  I expect that once I get it working and wrap my head around
it I’ll have some ideas that I’ll float around improving this area, some
use cases to propose that would possibly be external to the main
deployment, or additive. First thing first is understanding all the points
in doing it the hard way ;)


-- 

Sent with Airmail

On September 27, 2016 at 12:41:10, Nick Allen (n...@nickallen.org) wrote:

Hi Otto -

I would agree with you. We do not have documentation that describes how to
'permanently install' a new parser.. Your contribution would be highly
appreciated in this area.

With the Ansible-based deployment of today, most likely you will have to
touch some of Metron's Ansible source code. An alternative would be to
mimic portions of Metron's deployment code, and manage that in its own
project, which would deploy your new parser. But of course, if we can find
ways to make this task easier, we will.

You may not have to touch each of these areas, but they at least will
provide you with a better understanding of how everything is stitched
together.

*Monit *- The Monit integration lives in `metron-deployment/roles/monit`.
You can follow the pattern of
metron-deployment/roles/monit/templates/monit/parsers.monit to add your own
parser definition to Monit.

*Parsers* - The start script in
`metron-platform/metron-parsers/src/main/scripts/start_parser_topology.sh`
will give you good hooks into how each of the parsers are started.

*Setup* - There are various setup tasks for the streaming functionality
that live under `metron-deployment/roles/metron_streaming`. To understand
that process, start at `tasks/main.yml`.

I probably missed something, but let me know if you have questions.




On Tue, Sep 27, 2016 at 12:17 PM, Otto Fowler 
wrote:

> My wish, is that when I do an ansible-playbook -v -i {my configuration}
> metron_full_install.yml to my cluster - or do the full_dev-> vagrant that
> my parser / topology is deployed, started and monitored the same way as
the
> current bro, snort, and yaf parsers are.
>
> I might be misunderstanding something however. I seems to me that all the
> examples of adding other parsers are temporary and not permanent because
> they do not have the full deployment, kind of push the config and run the
> script and you are going. Am I missing something? Would the squid sample
> steps result in a parser topology that would survive restarts / reboots
> etc?
>
> On September 27, 2016 at 12:06:44, James Sirota (jsir...@apache.org)
> wrote:
>
> Just so I completely understand what you are asking for...you want to
know
> how to create a new parser topology with the JSON parser and plug it into
> Monit so you can monitor and restart it on demand?
>
> 27.09.2016, 09:03, "Otto Fowler" :
> > Thanks James,
> >
> > I want to deploy an instance of the JSONMapParser into my POC cluster
and
> vagrant. I’m trying to work out exactly how to add a new configured
parser
> instance to the deployment. I think these instructions would be a good
> extension to the squid stuff that is already there. If I could get that
> going and add a new parser all the way through, then maybe I can
contribute
> something in that area. The ability to do this will also enable some of
> the other work you mentioned.
> >
> > On September 27, 2016 at 11:51:41, James Sirota (jsir...@apache.org)
> wrote:
> >
> >> There are three types of parsers you can have currently. Our preferred
> way is to use Grok parser. The only thing you need to do there is to
define
> your Grok statement and the parser will uptake it and do the rest. That
is
> what most of our documentation reflect. The second type of parser that we
> have is a java parser, where you actually extend a parser class and
define
> your own custom parsing logic. We intend this type of parser for high
> velocity feeds that require custom parsing logic that is not easily
> attainable by Grok. The third type of parser is the one you have been
> working on, a Json parser. This is a parser designed to take pre-parsed
> JSON for sensors that either log in JSON format natively or have been
> pre-parsed for us by some system upstream.
> >>
> >> Parsers don't integrate with Monit by default. We can come up with
some
> instructions for you on how to do that.
> >>
> >> I should also note there are 2 additional parser types that are on the
> road map. METRON-295 (scripting bolt), which is a parser that allows you
to
> uptake something like javascript, lua, etc., for doing the parsing. There
> is also METRON-288, which is a XSL parser designed to parse XML
documents.
> If either of these are of interest to you we would welcome this
> contribution and we can work with you to get you 

[GitHub] incubator-metron issue #278: Metron 451 SerDeUtils - java.lang.ClassNotFound...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/278
  
The license check failed.  Take a look at the 
[`build_utils/README`](build_utils/README.md).  I believe we just need to 
update the `dependencies_with_url.csv` file with joda-time.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Complete steps to add a new parser

[Nick Allen]

A few other points of interest...

*Sample PCAP* - The sample pcap data that is replayed is installed by the
'sensor-test-mode' that lives at
`metron-deployment/roles/sensor-test-mode`.  If you don't want to use that,
just don't turn off 'sensor-test-mode'.

*ES Templates* - These live under
`metron-deployments/roles/metron_elasticsearch_templates`.  Imagine that. :)



On Tue, Sep 27, 2016 at 12:31 PM, Nick Allen  wrote:

> Hi Otto -
>
> I would agree with you.  We do not have documentation that describes how
> to 'permanently install' a new parser..  Your contribution would be highly
> appreciated in this area.
>
> With the Ansible-based deployment of today, most likely you will have to
> touch some of Metron's Ansible source code.  An alternative would be to
> mimic portions of Metron's deployment code, and manage that in its own
> project, which would deploy your new parser.  But of course, if we can find
> ways to make this task easier, we will.
>
> You may not have to touch each of these areas, but they at least will
> provide you with a better understanding of how everything is stitched
> together.
>
> *Monit *- The Monit integration lives in `metron-deployment/roles/monit`.
> You can follow the pattern of 
> metron-deployment/roles/monit/templates/monit/parsers.monit
> to add your own parser definition to Monit.
>
> *Parsers* - The start script in `metron-platform/metron-
> parsers/src/main/scripts/start_parser_topology.sh` will give you good
> hooks into how each of the parsers are started.
>
> *Setup* - There are various setup tasks for the streaming functionality
> that live under `metron-deployment/roles/metron_streaming`.  To
> understand that process, start at `tasks/main.yml`.
>
> I probably missed something, but let me know if you have questions.
>
>
>
>
> On Tue, Sep 27, 2016 at 12:17 PM, Otto Fowler 
> wrote:
>
>> My wish, is that when I do an ansible-playbook -v -i {my configuration}
>> metron_full_install.yml  to my cluster - or do the full_dev-> vagrant that
>> my parser / topology is deployed, started and monitored the same way as
>> the
>> current bro, snort, and yaf parsers are.
>>
>> I might be misunderstanding something however.  I seems to me that all the
>> examples of adding other parsers are temporary and not permanent because
>> they do not have the full deployment, kind of push the config and run the
>> script and you are going.  Am I missing something?  Would the squid sample
>> steps result in a parser topology that would survive restarts / reboots
>> etc?
>>
>> On September 27, 2016 at 12:06:44, James Sirota (jsir...@apache.org)
>> wrote:
>>
>> Just so I completely understand what you are asking for...you want to know
>> how to create a new parser topology with the JSON parser and plug it into
>> Monit so you can monitor and restart it on demand?
>>
>> 27.09.2016, 09:03, "Otto Fowler" :
>> > Thanks James,
>> >
>> > I want to deploy an instance of the JSONMapParser into my POC cluster
>> and
>> vagrant.  I’m trying to work out exactly how to add a new configured
>> parser
>> instance to the deployment.  I think these instructions would be a good
>> extension to the squid stuff that is already there.  If I could get that
>> going and add a new parser all the way through, then maybe I can
>> contribute
>> something in that area.  The ability to do this will also enable some of
>> the other work you mentioned.
>> >
>> > On September 27, 2016 at 11:51:41, James Sirota (jsir...@apache.org)
>> wrote:
>> >
>> >> There are three types of parsers you can have currently. Our preferred
>> way is to use Grok parser. The only thing you need to do there is to
>> define
>> your Grok statement and the parser will uptake it and do the rest. That is
>> what most of our documentation reflect. The second type of parser that we
>> have is a java parser, where you actually extend a parser class and define
>> your own custom parsing logic. We intend this type of parser for high
>> velocity feeds that require custom parsing logic that is not easily
>> attainable by Grok. The third type of parser is the one you have been
>> working on, a Json parser. This is a parser designed to take pre-parsed
>> JSON for sensors that either log in JSON format natively or have been
>> pre-parsed for us by some system upstream.
>> >>
>> >> Parsers don't integrate with Monit by default. We can come up with some
>> instructions for you on how to do that.
>> >>
>> >> I should also note there are 2 additional parser types that are on the
>> road map. METRON-295 (scripting bolt), which is a parser that allows you
>> to
>> uptake something like javascript, lua, etc., for doing the parsing. There
>> is also METRON-288, which is a XSL parser designed to parse XML documents.
>> If either of these are of interest to you we would welcome this
>> contribution and we can work with you to get you started.
>> >>
>> >> 26.09.2016, 10:35, "Otto Fowler" 

Re: Complete steps to add a new parser

[Nick Allen]

Hi Otto -

I would agree with you.  We do not have documentation that describes how to
'permanently install' a new parser..  Your contribution would be highly
appreciated in this area.

With the Ansible-based deployment of today, most likely you will have to
touch some of Metron's Ansible source code.  An alternative would be to
mimic portions of Metron's deployment code, and manage that in its own
project, which would deploy your new parser.  But of course, if we can find
ways to make this task easier, we will.

You may not have to touch each of these areas, but they at least will
provide you with a better understanding of how everything is stitched
together.

*Monit *- The Monit integration lives in `metron-deployment/roles/monit`.
You can follow the pattern of
metron-deployment/roles/monit/templates/monit/parsers.monit to add your own
parser definition to Monit.

*Parsers* - The start script in
`metron-platform/metron-parsers/src/main/scripts/start_parser_topology.sh`
will give you good hooks into how each of the parsers are started.

*Setup* - There are various setup tasks for the streaming functionality
that live under `metron-deployment/roles/metron_streaming`.  To understand
that process, start at `tasks/main.yml`.

I probably missed something, but let me know if you have questions.




On Tue, Sep 27, 2016 at 12:17 PM, Otto Fowler 
wrote:

> My wish, is that when I do an ansible-playbook -v -i {my configuration}
> metron_full_install.yml  to my cluster - or do the full_dev-> vagrant that
> my parser / topology is deployed, started and monitored the same way as the
> current bro, snort, and yaf parsers are.
>
> I might be misunderstanding something however.  I seems to me that all the
> examples of adding other parsers are temporary and not permanent because
> they do not have the full deployment, kind of push the config and run the
> script and you are going.  Am I missing something?  Would the squid sample
> steps result in a parser topology that would survive restarts / reboots
> etc?
>
> On September 27, 2016 at 12:06:44, James Sirota (jsir...@apache.org)
> wrote:
>
> Just so I completely understand what you are asking for...you want to know
> how to create a new parser topology with the JSON parser and plug it into
> Monit so you can monitor and restart it on demand?
>
> 27.09.2016, 09:03, "Otto Fowler" :
> > Thanks James,
> >
> > I want to deploy an instance of the JSONMapParser into my POC cluster and
> vagrant.  I’m trying to work out exactly how to add a new configured parser
> instance to the deployment.  I think these instructions would be a good
> extension to the squid stuff that is already there.  If I could get that
> going and add a new parser all the way through, then maybe I can contribute
> something in that area.  The ability to do this will also enable some of
> the other work you mentioned.
> >
> > On September 27, 2016 at 11:51:41, James Sirota (jsir...@apache.org)
> wrote:
> >
> >> There are three types of parsers you can have currently. Our preferred
> way is to use Grok parser. The only thing you need to do there is to define
> your Grok statement and the parser will uptake it and do the rest. That is
> what most of our documentation reflect. The second type of parser that we
> have is a java parser, where you actually extend a parser class and define
> your own custom parsing logic. We intend this type of parser for high
> velocity feeds that require custom parsing logic that is not easily
> attainable by Grok. The third type of parser is the one you have been
> working on, a Json parser. This is a parser designed to take pre-parsed
> JSON for sensors that either log in JSON format natively or have been
> pre-parsed for us by some system upstream.
> >>
> >> Parsers don't integrate with Monit by default. We can come up with some
> instructions for you on how to do that.
> >>
> >> I should also note there are 2 additional parser types that are on the
> road map. METRON-295 (scripting bolt), which is a parser that allows you to
> uptake something like javascript, lua, etc., for doing the parsing. There
> is also METRON-288, which is a XSL parser designed to parse XML documents.
> If either of these are of interest to you we would welcome this
> contribution and we can work with you to get you started.
> >>
> >> 26.09.2016, 10:35, "Otto Fowler" :
> >>> Are all the steps required to add a parser documented anywhere? The
> squid
> >>> document starts the topology, but I don’t think that integrates it in
> with
> >>> monit for example. Or does that actually happen?
> >>
> >> ---
> >> Thank you,
> >>
> >> James Sirota
> >> PPMC- Apache Metron (Incubating)
> >> jsirota AT apache DOT org
>
> ---
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>



-- 
Nick Allen 

Re: Complete steps to add a new parser

[Otto Fowler]

I am also going to create the ES Template and possibly modify or create a
new dashboard.  Full peer.

On September 27, 2016 at 12:17:23, Otto Fowler (ottobackwa...@gmail.com)
wrote:

My wish, is that when I do an ansible-playbook -v -i {my configuration}
metron_full_install.yml  to my cluster - or do the full_dev-> vagrant that
my parser / topology is deployed, started and monitored the same way as the
current bro, snort, and yaf parsers are.

I might be misunderstanding something however.  I seems to me that all the
examples of adding other parsers are temporary and not permanent because
they do not have the full deployment, kind of push the config and run the
script and you are going.  Am I missing something?  Would the squid sample
steps result in a parser topology that would survive restarts / reboots etc?

On September 27, 2016 at 12:06:44, James Sirota (jsir...@apache.org) wrote:

Just so I completely understand what you are asking for...you want to know
how to create a new parser topology with the JSON parser and plug it into
Monit so you can monitor and restart it on demand?

27.09.2016, 09:03, "Otto Fowler" :
> Thanks James,
>
> I want to deploy an instance of the JSONMapParser into my POC cluster and
vagrant.  I’m trying to work out exactly how to add a new configured parser
instance to the deployment.  I think these instructions would be a good
extension to the squid stuff that is already there.  If I could get that
going and add a new parser all the way through, then maybe I can contribute
something in that area.  The ability to do this will also enable some of
the other work you mentioned.
>
> On September 27, 2016 at 11:51:41, James Sirota (jsir...@apache.org)
wrote:
>
>> There are three types of parsers you can have currently. Our preferred
way is to use Grok parser. The only thing you need to do there is to define
your Grok statement and the parser will uptake it and do the rest. That is
what most of our documentation reflect. The second type of parser that we
have is a java parser, where you actually extend a parser class and define
your own custom parsing logic. We intend this type of parser for high
velocity feeds that require custom parsing logic that is not easily
attainable by Grok. The third type of parser is the one you have been
working on, a Json parser. This is a parser designed to take pre-parsed
JSON for sensors that either log in JSON format natively or have been
pre-parsed for us by some system upstream.
>>
>> Parsers don't integrate with Monit by default. We can come up with some
instructions for you on how to do that.
>>
>> I should also note there are 2 additional parser types that are on the
road map. METRON-295 (scripting bolt), which is a parser that allows you to
uptake something like javascript, lua, etc., for doing the parsing. There
is also METRON-288, which is a XSL parser designed to parse XML documents.
If either of these are of interest to you we would welcome this
contribution and we can work with you to get you started.
>>
>> 26.09.2016, 10:35, "Otto Fowler" :
>>> Are all the steps required to add a parser documented anywhere? The
squid
>>> document starts the topology, but I don’t think that integrates it in
with
>>> monit for example. Or does that actually happen?
>>
>> ---
>> Thank you,
>>
>> James Sirota
>> PPMC- Apache Metron (Incubating)
>> jsirota AT apache DOT org

---
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Re: Complete steps to add a new parser

[James Sirota]

Just so I completely understand what you are asking for...you want to know how 
to create a new parser topology with the JSON parser and plug it into Monit so 
you can monitor and restart it on demand?

27.09.2016, 09:03, "Otto Fowler" :
> Thanks James,
>
> I want to deploy an instance of the JSONMapParser into my POC cluster and 
> vagrant.  I’m trying to work out exactly how to add a new configured parser 
> instance to the deployment.  I think these instructions would be a good 
> extension to the squid stuff that is already there.  If I could get that 
> going and add a new parser all the way through, then maybe I can contribute 
> something in that area.  The ability to do this will also enable some of the 
> other work you mentioned.
>
> On September 27, 2016 at 11:51:41, James Sirota (jsir...@apache.org) wrote:
>
>> There are three types of parsers you can have currently. Our preferred way 
>> is to use Grok parser. The only thing you need to do there is to define your 
>> Grok statement and the parser will uptake it and do the rest. That is what 
>> most of our documentation reflect. The second type of parser that we have is 
>> a java parser, where you actually extend a parser class and define your own 
>> custom parsing logic. We intend this type of parser for high velocity feeds 
>> that require custom parsing logic that is not easily attainable by Grok. The 
>> third type of parser is the one you have been working on, a Json parser. 
>> This is a parser designed to take pre-parsed JSON for sensors that either 
>> log in JSON format natively or have been pre-parsed for us by some system 
>> upstream.
>>
>> Parsers don't integrate with Monit by default. We can come up with some 
>> instructions for you on how to do that.
>>
>> I should also note there are 2 additional parser types that are on the road 
>> map. METRON-295 (scripting bolt), which is a parser that allows you to 
>> uptake something like javascript, lua, etc., for doing the parsing. There is 
>> also METRON-288, which is a XSL parser designed to parse XML documents. If 
>> either of these are of interest to you we would welcome this contribution 
>> and we can work with you to get you started.
>>
>> 26.09.2016, 10:35, "Otto Fowler" :
>>> Are all the steps required to add a parser documented anywhere? The squid
>>> document starts the topology, but I don’t think that integrates it in with
>>> monit for example. Or does that actually happen?
>>
>> ---
>> Thank you,
>>
>> James Sirota
>> PPMC- Apache Metron (Incubating)
>> jsirota AT apache DOT org

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Re: [DISCUSS] Upcoming Metron Build

[Michael Miklavcic]

Also need this Jira to bump the version.

https://issues.apache.org/jira/browse/METRON-398

On Tue, Sep 27, 2016 at 11:38 AM, James Sirota  wrote:

> We are preparing the next release (will be put up for a vote within the
> next few days) with the following list of Jiras.  Do you feel there are any
> other Jiras that should go into this release or are there any critical bugs
> that anyone knows of that we should address before releasing?  Please
> comment on this thread.  Otherwise, we will put up the release for a vote
> shortly.
>
> METRON-457 Correct GrokParser logging spelling error (mmiklavc via
> cestella) closes apache/incubator-metron#274
> METRON-449 JSONMapParser should unfold maps to arbitrary depths closes
> apache/incubator-metron#271
> METRON-453: Add a stellar shell function to open an external editor
> and return the editor's contents closes apache/incubator-metron#272
> METRON-452: Add rudimentary configuration management functions to
> Stellar closes apache/incubator-metron#269
> METRON-374: Add appropriate bundled 3rd party licenses to NOTICE and
> LICENSE where appropriate closes apache/incubator-metron#229
> METRON-427 Create Ambari Management Pack for Metron Installation
> closes apache/incubator-metron#266
> METRON-434: JSON Parser closes apache/incubator-metron#261
> METRON-437 Profile Definition's 'inputTopic' field is Extraneous
> (nickwallen) closes apache/incubator-metron#264
> METRON-445 Fix typos in metron-deployment roles (JonZeolla via
> nickwallen) closes apache/incubator-metron#267
> METRON-438: Back the Stellar REPL with a readline implementation
> closes apache/incubator-metron#265
> METRON-426: Stellar does not support scientific notation as a literal
> closes apache/incubator-metron#257
> METRON-435: Create Stellar REPL (nickwallen via cestella) closes
> apache/incubator-metron#262
> METRON-436: Updated architecture diagrams for Metron READMEs
> (anandsubbu via cestella) closes apache/incubator-metron#263
> METRON-433: Documentation update closes apache/incubator-metron#260
> METRON-428: Allow a kafka offset to be passed to the ParserTopology
> CLI closes apache/incubator-metron#258
> METRON-429 Profiler Missing Dependencies When Uber Jar Deployed
> (nickwallen) closes apache/incubator-metron#259
> METRON-384  Deployment fails at task Wait for Elasticsearch Host
> to Start (2xyo via dlyle65535) closes apache/incubator-metron#221
> METRON-257 Enable pcap result pagination from the Pcap CLI (mmiklavc
> via cestella) closes apache/incubator-metron#256
> METRON-420 Add Expiration to a Profile Definition (nickwallen) closes
> apache/incubator-metron#254
> METRON-413 Allow Start/End Time Range Search in Profiler Client API
> (nickwallen) closes apache/incubator-metron#249
> METRON-419 Update Tuple to HBase Mapper/Bolt to Set TTL (nickwallen)
> closes apache/incubator-metron#252
> METRON-422 Remove bluecoat.json (cestella via nickwallen) closes
> apache/incubator-metron#255
> METRON-418 Set TTL on HBase Puts (nickwallen) closes
> apache/incubator-metron#251
> METRON-415: Allow a Profile to Store Any Type as its Value closes
> apache/incubator-metron#253
> METRON-411 Support Greater Range of Profile Periods  (nickwallen)
> closes apache/incubator-metron#246
> METRON-416: Provide the ability to store mergeable data structures for
> summarizing data on-line closes apache/incubator-metron#250
> METRON-397: Add a stellar function to interact with the HBase
> enrichment table closes apache/incubator-metron#234
> METRON-391 Create Stellar Function to Read Profile Data for Model
> Scoring (nickwallen via cestella) closes apache/incubator-metron#242
> METRON-414 Kibana Ansible Install Fails with SSL Error closes
> apache/incubator-metron#248
> METRON-407: We currently do not provide defaults if the Stix
> Observable does not specify a condition closes apache/incubator-metron#244
> METRON-406: Stellar variable resolution does not resolve variables
> with ':' in them closes apache/incubator-metron#243
> METRON-399 Stellar Date Functions Should Default to Current Time
> (nickwallen) closes apache/incubator-metron#237
> METRON-389 Create Java API to Read Profile Data During Model Scoring
> (nickwallen) closes apache/incubator-metron#236
> METRON-385 Create Ambari Service Definition for Indexing (justinleet
> via cestella) closes apache/incubator-metron#222
> METRON-395 Fix Metron Bro parser not parsing some timestamp values
> (mmiklavc via cestella) closes apache/incubator-metron#232
> METRON-400 Deploy Probes to running Docker Container closes
> apache/incubator-metron#241
> METRON-408 Intermittent Failures of Profile Integration Tests
> (nickwallen via cestella) closes apache/incubator-metron#245
> METRON-381: Add support for multiple reducers in pcap_query.sh closes
> apache/incubator-metron#217
> METRON-404 

[DISCUSS] Upcoming Metron Build

[James Sirota]

We are preparing the next release (will be put up for a vote within the next 
few days) with the following list of Jiras.  Do you feel there are any other 
Jiras that should go into this release or are there any critical bugs that 
anyone knows of that we should address before releasing?  Please comment on 
this thread.  Otherwise, we will put up the release for a vote shortly.  

METRON-457 Correct GrokParser logging spelling error (mmiklavc via 
cestella) closes apache/incubator-metron#274
METRON-449 JSONMapParser should unfold maps to arbitrary depths closes 
apache/incubator-metron#271
METRON-453: Add a stellar shell function to open an external editor and 
return the editor's contents closes apache/incubator-metron#272
METRON-452: Add rudimentary configuration management functions to Stellar 
closes apache/incubator-metron#269
METRON-374: Add appropriate bundled 3rd party licenses to NOTICE and 
LICENSE where appropriate closes apache/incubator-metron#229
METRON-427 Create Ambari Management Pack for Metron Installation closes 
apache/incubator-metron#266
METRON-434: JSON Parser closes apache/incubator-metron#261
METRON-437 Profile Definition's 'inputTopic' field is Extraneous 
(nickwallen) closes apache/incubator-metron#264
METRON-445 Fix typos in metron-deployment roles (JonZeolla via nickwallen) 
closes apache/incubator-metron#267
METRON-438: Back the Stellar REPL with a readline implementation closes 
apache/incubator-metron#265
METRON-426: Stellar does not support scientific notation as a literal 
closes apache/incubator-metron#257
METRON-435: Create Stellar REPL (nickwallen via cestella) closes 
apache/incubator-metron#262
METRON-436: Updated architecture diagrams for Metron READMEs (anandsubbu 
via cestella) closes apache/incubator-metron#263
METRON-433: Documentation update closes apache/incubator-metron#260
METRON-428: Allow a kafka offset to be passed to the ParserTopology CLI 
closes apache/incubator-metron#258
METRON-429 Profiler Missing Dependencies When Uber Jar Deployed 
(nickwallen) closes apache/incubator-metron#259
METRON-384  Deployment fails at task Wait for Elasticsearch Host to 
Start (2xyo via dlyle65535) closes apache/incubator-metron#221
METRON-257 Enable pcap result pagination from the Pcap CLI (mmiklavc via 
cestella) closes apache/incubator-metron#256
METRON-420 Add Expiration to a Profile Definition (nickwallen) closes 
apache/incubator-metron#254
METRON-413 Allow Start/End Time Range Search in Profiler Client API 
(nickwallen) closes apache/incubator-metron#249
METRON-419 Update Tuple to HBase Mapper/Bolt to Set TTL (nickwallen) closes 
apache/incubator-metron#252
METRON-422 Remove bluecoat.json (cestella via nickwallen) closes 
apache/incubator-metron#255
METRON-418 Set TTL on HBase Puts (nickwallen) closes 
apache/incubator-metron#251
METRON-415: Allow a Profile to Store Any Type as its Value closes 
apache/incubator-metron#253
METRON-411 Support Greater Range of Profile Periods  (nickwallen) closes 
apache/incubator-metron#246
METRON-416: Provide the ability to store mergeable data structures for 
summarizing data on-line closes apache/incubator-metron#250
METRON-397: Add a stellar function to interact with the HBase enrichment 
table closes apache/incubator-metron#234
METRON-391 Create Stellar Function to Read Profile Data for Model Scoring 
(nickwallen via cestella) closes apache/incubator-metron#242
METRON-414 Kibana Ansible Install Fails with SSL Error closes 
apache/incubator-metron#248
METRON-407: We currently do not provide defaults if the Stix Observable 
does not specify a condition closes apache/incubator-metron#244
METRON-406: Stellar variable resolution does not resolve variables with ':' 
in them closes apache/incubator-metron#243
METRON-399 Stellar Date Functions Should Default to Current Time 
(nickwallen) closes apache/incubator-metron#237
METRON-389 Create Java API to Read Profile Data During Model Scoring 
(nickwallen) closes apache/incubator-metron#236
METRON-385 Create Ambari Service Definition for Indexing (justinleet via 
cestella) closes apache/incubator-metron#222
METRON-395 Fix Metron Bro parser not parsing some timestamp values 
(mmiklavc via cestella) closes apache/incubator-metron#232
METRON-400 Deploy Probes to running Docker Container closes 
apache/incubator-metron#241
METRON-408 Intermittent Failures of Profile Integration Tests (nickwallen 
via cestella) closes apache/incubator-metron#245
METRON-381: Add support for multiple reducers in pcap_query.sh closes 
apache/incubator-metron#217
METRON-404 Stellar Compiler Hides Function Initialization Errors 
(nickwallen) closes apache/incubator-metron#240
METRON-392 Allow User to Define Custom 'Group By' for a Profile 
(nickwallen) closes apache/incubator-metron#230
METRON-383: Create Ambari Service Definition for Metron Parsers 

[GitHub] incubator-metron issue #275: METRON-459 Bad file location for org.apache.met...

[danieljue]

Github user danieljue commented on the issue:

https://github.com/apache/incubator-metron/pull/275
  
Sorry, while fixing 451 I neglected to put 451's change in it's own branch. 
 Reverted the 451 fix from this PR and then cherry picked 451's fix into it's 
own branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #268: METRON-421 Make Stellar Profiler Client ...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/268#discussion_r80693264
  
--- Diff: 
metron-analytics/metron-profiler-client/src/test/resources/core-site.xml ---
@@ -0,0 +1,99 @@
+  
+
+
+  fs.defaultFS
+  hdfs://node1:8020
--- End diff --

I think this is the cause of the unit test failures.  We really shouldn't 
have a core-site.xml here.  What's caused it to be required?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #275: METRON-459 Bad file location for org.apache.met...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/275
  
+1 - Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #271: METRON-449 JSONMapParser should unfold m...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/271


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #271: METRON-449 JSONMapParser should unfold maps to ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/271
  
+1, tested in single node vagrant.  

Did the following:
* Created a config called `custom` at 
`$METRON_HOME/config/zookeeper/parsers/custom.json`:
```
{
  "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
  "sensorTopic":"custom",
  "parserConfig" : {
"mapStrategy" : "UNFOLD"
  }
}
```
* Created kafka topic for `custom` via 
`/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 
--create --topic custom --partitions 1 --replication-factor 1`
* Loaded the configs into zookeeper via 
`$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -i 
/usr/metron/0.2.0BETA/config/zookeeper -z node1:2181`
* Started the topology via 
`/usr/metron/0.2.0BETA/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 
-s custom`
* Created some sample data in `~/test_data.dat`:
```
{ "foo" : "bar" }
{ "map" : { "blah" : 1 } }
{ "foo" : "grok", "map" : { "blah" : { "tim" : "sort" } } }
```
* Loaded the sample data into the kafka queue via `cat ~/test_data.dat | 
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list 
node1:6667 --topic custom`
* Checked in ES Head that I had a message with a field called `map:blah:tim`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---