Re: [DISCUSS] Recurring community meetings to demo Metron features

[Kyle Richardson]

Great idea. +1 on the agenda. Maybe half demo of latest features / half
discussion on upcoming changes and new ideas.

-Kyle

On Wed, Sep 21, 2016 at 4:03 PM, zeo...@gmail.com  wrote:

> I'm in from CMU.  Zoom and WebEx work well.
>
> Only suggestion would be a basic agenda (I.e. feature list) prior to the
> meeting so people can do their homework.  Prior meaning ~24 hours before at
> a minimum IMO.
>
> Jon
>
> On Wed, Sep 21, 2016, 15:53 Tseytlin, Keren  >
> wrote:
>
> > Hi James,
> >
> > Kevin and I (and perhaps a couple others from our team) will join in too.
> > Keep us posted with the meeting details.
> >
> > Best,
> > Keren
> >
> > On 9/21/16, 2:26 PM, "Otto Fowler"  wrote:
> >
> > >Hi James,
> > >
> > >I think this is a great idea.  Zoom seems to work pretty well.
> > >
> > >On September 21, 2016 at 14:18:54, James Sirota (jsir...@apache.org)
> > >wrote:
> > >
> > >I want to setup recurring meetings that run twice a month where we can
> > >demo
> > >latest features of Metron and fixes for Metron . I want to have the
> first
> > >one of these this Friday on Sept. 23, 11AM PST.
> > >
> > >Would anyone object to doing this? If not, what medium do we want to do
> it
> > >in? I would like people to be able to use screen share and not just IRC.
> > >On
> > >my end I can provide a Zoom or a WebEx to facilitate this. Then we can
> > >post
> > >meeting notes onto the Metron boards and post the video up on you tube
> and
> > >link to it. Any other preferences/suggestions on how to do this?
> > >
> > >
> > >
> > >---
> > >Thank you,
> > >
> > >James Sirota
> > >PPMC- Apache Metron (Incubating)
> > >jsirota AT apache DOT org
> >
> > 
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
> > --
>
> Jon
>

[GitHub] incubator-metron issue #264: METRON-437 The Profile Definition's inputTopic ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/264
  
+1 by inspection


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: log parsers-

[Satish Abburi]

All, I have put together few interesting log sources what we are looking
and also mapped the existing Metron-JIRA#¹s for few of them.

https://drive.google.com/open?id=0B3HLRtVIDxauS3E3dE9mb1R3M2M

Also, attached same to the email.


Thanks,
Satish

On 9/14/16, 4:09 PM, "Satish Abburi"  wrote:

>
>Great Kyle! If you can make it by next Friday, that will be very helpful.
>
>I see BlueCoat is also in progress from Jira, any input on the current
>status?
>
>On 9/14/16, 4:06 PM, "Kyle Richardson"  wrote:
>
>>I have a working code for the ASA piece (METRON-363). Just finishing up
>>some edge case testing. I'll submit a PR for it within your 2 week
>>timeframe.
>>
>>Thanks,
>>Kyle
>>
>>> On Sep 14, 2016, at 6:58 PM, Satish Abburi 
>>>wrote:
>>> 
>>> 
>>> Thanks, timelines are 2 weeks from now. Thanks.
>>> 
>>> From: Poornima Ravindra Mulukutla
>>>>
>>> Reply-To: 
>>>"u...@metron.incubator.apache.org>>g
" 
>
>>> Date: Wednesday, September 14, 2016 at 3:26 PM
>>> To: 
>>>"u...@metron.incubator.apache.org>>g
" 
>
>>> Cc: 
>>>"dev@metron.incubator.apache.org
>>>"
>>> 
>>>

>>> Subject: Re: log parsers-
>>> 
>>> Thank you
>>> 
>>> I am happy to take up ASA log file analyser, what is the timeline you
>>>are looking for so that I will plan accordingly?
>>> 
>>> In the past I have done BlueCoat log analyser when I was doing research
>>>on HTTP specification (published a patent has created big change in HTTP
>>>designs), recently adopted for the Microsoft IE 11
>>> 
>>> On Wed, Sep 14, 2016 at 6:54 PM, Satish Abburi
>>>> wrote:
>>> 
>>> Hi, we are trying to build parsers for our Phase1 demo on Metron
>>>platform. Would like to find, if anyone already has these parsers
>>>developed.
>>> We already started working on  Windows parser, rest planning to start
>>>this week. We can leverage if some thing avaialble or collaborate
>>>appropriately.
>>> 
>>> 
>>>  *   ASA (Firewall) Metron-363
>>>  *   Windows (Desktop) - METRON-165
>>>  *   Unix (OS) Metron-175
>>>  *   Email
>>>  *   BlueCoat(Proxy) METRON-162
>>> 
>>> Thanks for your help!
>>> Satish
>>> 
>



LogParsers.xlsx
Description: LogParsers.xlsx

Re: [DISCUSS] Metron standard field names

[Yohann Lepage]

2016-09-21 22:00 GMT+02:00 zeo...@gmail.com :
> Elasticsearch can't use periods in field names,
It's possible again since the latest release
https://www.elastic.co/blog/elasticsearch-2-4-0-released


-- 
Yohann L.

Re: [DISCUSS] Recurring community meetings to demo Metron features

[zeo...@gmail.com]

I'm in from CMU.  Zoom and WebEx work well.

Only suggestion would be a basic agenda (I.e. feature list) prior to the
meeting so people can do their homework.  Prior meaning ~24 hours before at
a minimum IMO.

Jon

On Wed, Sep 21, 2016, 15:53 Tseytlin, Keren 
wrote:

> Hi James,
>
> Kevin and I (and perhaps a couple others from our team) will join in too.
> Keep us posted with the meeting details.
>
> Best,
> Keren
>
> On 9/21/16, 2:26 PM, "Otto Fowler"  wrote:
>
> >Hi James,
> >
> >I think this is a great idea.  Zoom seems to work pretty well.
> >
> >On September 21, 2016 at 14:18:54, James Sirota (jsir...@apache.org)
> >wrote:
> >
> >I want to setup recurring meetings that run twice a month where we can
> >demo
> >latest features of Metron and fixes for Metron . I want to have the first
> >one of these this Friday on Sept. 23, 11AM PST.
> >
> >Would anyone object to doing this? If not, what medium do we want to do it
> >in? I would like people to be able to use screen share and not just IRC.
> >On
> >my end I can provide a Zoom or a WebEx to facilitate this. Then we can
> >post
> >meeting notes onto the Metron boards and post the video up on you tube and
> >link to it. Any other preferences/suggestions on how to do this?
> >
> >
> >
> >---
> >Thank you,
> >
> >James Sirota
> >PPMC- Apache Metron (Incubating)
> >jsirota AT apache DOT org
>
> 
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
> --

Jon

Re: [DISCUSS] Metron standard field names

[zeo...@gmail.com]

Elasticsearch can't use periods in field names, I think that's part of why
they aren't used generally.  I think this is a worthwhile discussion
though, specifically regarding the timestamp and protocol discussion you
started above.

On Wed, Sep 21, 2016, 15:52 Yohann Lepage  wrote:

> Hi everyone,
>
> I wanted to solicit some discussion around  Metron standard field names.
>
> I would love to have "convenient" field names. As convenient, I mean:
> short, not ambiguous, well-known, documented.
>
> Here is my feeling regarding  the actual standard field names[0]:
> - ip_src_addr:  too long, could be "ip.src"
> - ip_dst_addr:   too long, could be "ip.dst"
> - ip_src_port:  I prefer tcp.srcport
> - ip_dst_port: I prefer tcp.dstport
> - protocol:  ambiguous, I prefer ip.proto
> - timestamp: ambiguous, which timetamp ? When the event happend  ? Or
> when the log has been received ?  Or when the log has been generated ?
>
> Also, it would be useful to publish a "reference list" of  fields
> (It's helpful when you write correlation rules or new parsers).
> Example: https://www.wireshark.org/docs/dfref/
>
> Wireshark, as a well-known tool used by SOC investigator,  could be a
> good starting point[1][2] to find convenient field names. Quick
> example (markdown format):
>
> |Field Name|Description |Type|Example|
> |---|---|---|---|
> |frame.len| Frame length on the wire|   Unsigned integer, 4 bytes|123|
> |ip.src|Source Address|IPv4 address|192.0.2.1|
> |ip.dst|Destination Address|IPv4 address|192.0.2.1|
> |ip.addr|Source or Destination Address|IPv4 address|192.0.2.1|
> |ip.proto|Protocol|Unsigned integer, 1 byte|6|
> |ip.srcport|UDP or TCP source Port|Unsigned integer, 2 bytes|12345|
> |ip.dstport|UDP or TCP destination Port|Unsigned integer, 2 bytes|53|
> |tcp.srcport|TCP source Port|Unsigned integer, 2 bytes|12345|
> |tcp.dstport|TCP destination Port|Unsigned integer, 2 bytes|80|
> |tcp.len|TCP Segment Length|Unsigned integer, 4 bytes|123|
> |udp.srcport|UDP source Port|Unsigned integer, 2 bytes|12345|
> |udp.dstport|UDP destination Port|Unsigned integer, 2 bytes|67|
> |http.host|Host|Character string|example.net|
> |http.referer|Referer|Character string|http://example.net/foo|
> |http.request.method|Request Method|Character string|POST|
> |http.full_uri or uri |Full request URI|Character
> string|http://example.net/foo?bar=baz|
>
> What do you think?
>
> My 2 cents,
> --
> Yohann L.
>
-- 

Jon

[DISCUSS] Metron standard field names

[Yohann Lepage]

Hi everyone,

I wanted to solicit some discussion around  Metron standard field names.

I would love to have "convenient" field names. As convenient, I mean:
short, not ambiguous, well-known, documented.

Here is my feeling regarding  the actual standard field names[0]:
- ip_src_addr:  too long, could be "ip.src"
- ip_dst_addr:   too long, could be "ip.dst"
- ip_src_port:  I prefer tcp.srcport
- ip_dst_port: I prefer tcp.dstport
- protocol:  ambiguous, I prefer ip.proto
- timestamp: ambiguous, which timetamp ? When the event happend  ? Or
when the log has been received ?  Or when the log has been generated ?

Also, it would be useful to publish a "reference list" of  fields
(It's helpful when you write correlation rules or new parsers).
Example: https://www.wireshark.org/docs/dfref/

Wireshark, as a well-known tool used by SOC investigator,  could be a
good starting point[1][2] to find convenient field names. Quick
example (markdown format):

|Field Name|Description |Type|Example|
|---|---|---|---|
|frame.len| Frame length on the wire|   Unsigned integer, 4 bytes|123|
|ip.src|Source Address|IPv4 address|192.0.2.1|
|ip.dst|Destination Address|IPv4 address|192.0.2.1|
|ip.addr|Source or Destination Address|IPv4 address|192.0.2.1|
|ip.proto|Protocol|Unsigned integer, 1 byte|6|
|ip.srcport|UDP or TCP source Port|Unsigned integer, 2 bytes|12345|
|ip.dstport|UDP or TCP destination Port|Unsigned integer, 2 bytes|53|
|tcp.srcport|TCP source Port|Unsigned integer, 2 bytes|12345|
|tcp.dstport|TCP destination Port|Unsigned integer, 2 bytes|80|
|tcp.len|TCP Segment Length|Unsigned integer, 4 bytes|123|
|udp.srcport|UDP source Port|Unsigned integer, 2 bytes|12345|
|udp.dstport|UDP destination Port|Unsigned integer, 2 bytes|67|
|http.host|Host|Character string|example.net|
|http.referer|Referer|Character string|http://example.net/foo|
|http.request.method|Request Method|Character string|POST|
|http.full_uri or uri |Full request URI|Character
string|http://example.net/foo?bar=baz|

What do you think?

My 2 cents,
-- 
Yohann L.

Re: [DISCUSS] Recurring community meetings to demo Metron features

[Tseytlin, Keren]

Hi James,

Kevin and I (and perhaps a couple others from our team) will join in too.
Keep us posted with the meeting details.

Best,
Keren

On 9/21/16, 2:26 PM, "Otto Fowler"  wrote:

>Hi James,
>
>I think this is a great idea.  Zoom seems to work pretty well.
>
>On September 21, 2016 at 14:18:54, James Sirota (jsir...@apache.org)
>wrote:
>
>I want to setup recurring meetings that run twice a month where we can
>demo
>latest features of Metron and fixes for Metron . I want to have the first
>one of these this Friday on Sept. 23, 11AM PST.
>
>Would anyone object to doing this? If not, what medium do we want to do it
>in? I would like people to be able to use screen share and not just IRC.
>On
>my end I can provide a Zoom or a WebEx to facilitate this. Then we can
>post
>meeting notes onto the Metron boards and post the video up on you tube and
>link to it. Any other preferences/suggestions on how to do this?
>
>
>
>---
>Thank you,
>
>James Sirota
>PPMC- Apache Metron (Incubating)
>jsirota AT apache DOT org



The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.

[GitHub] incubator-metron pull request #257: METRON-426: Stellar does not support sci...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/257


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Recurring community meetings to demo Metron features

[Otto Fowler]

Hi James,

I think this is a great idea.  Zoom seems to work pretty well.

On September 21, 2016 at 14:18:54, James Sirota (jsir...@apache.org) wrote:

I want to setup recurring meetings that run twice a month where we can demo
latest features of Metron and fixes for Metron . I want to have the first
one of these this Friday on Sept. 23, 11AM PST.

Would anyone object to doing this? If not, what medium do we want to do it
in? I would like people to be able to use screen share and not just IRC. On
my end I can provide a Zoom or a WebEx to facilitate this. Then we can post
meeting notes onto the Metron boards and post the video up on you tube and
link to it. Any other preferences/suggestions on how to do this?



---
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Re: [DISCUSS] Ambari Integration

[Justin Leet]

We could definitely replace some of it, but have not replaced anything for
this PR. Most changes in the PR are in metron-deployment/packaging/ambari/
+ some light surrounding work to make some stuff available that wasn't.
The Ansible stuff is basically untouched if not actually untouched.
Actually provisioning the base cluster is a separate activity that will
live alongside the install. What we want as an end state is definitely
something I'd love to hear discussion on, and one I'd expect to see more of
as we build on this.

For a little more context, basically the mpack moves node assignment,
managing services, configuration, etc. to Ambari.  Basically the goal is to
just make everything a lot easier to install and configure in the general
case, which is a pain point that I've heard repeatedly.  It's (relatively)
easy to spin up on AWS because there's Ansible for it, but spinning up
random machines and putting Metron on top is more painful.

If you've gone through the regular Ambari install of a service (e.g.
through the UI), it's basically
1) Where do you want to install components?
2) What configs do you want on them?
3) Install and start them up?

The mpack is just setting up Ambari to make doing that easier by defining
the services, exposing some of the configuration (and I'm sure there's more
that should be exposed).

Justin

On Wed, Sep 21, 2016 at 1:53 PM, Otto Fowler 
wrote:

> Thanks Justin,
>
> So this should just replace what is currently happening if you do the full
> deployment, but you have not tested it as such?
> I think the difference in the ASW deployment that I saw was how it set the
> nodes to roles through the script.  Sorry if I overstated it.
>
>
> On September 21, 2016 at 13:45:14, Justin Leet (justinjl...@gmail.com)
> wrote:
>
> Hi Otto,
>
> Couple things to dig into a bit.  Let me know if I stray off what your
> question is, but I think this should give you the answer.
>
> For the mpack, it's just taking a cluster without Metron and turning it
> into a cluster running Metron (regardless of the cluster itself was
> provisioned).  I wasn't clear about it in my last message, but testing on
> AWS wasn't really about making sure small_cluster configuration was
> compatible with the mpack changes.  It was about making sure that we could
> go from a cluster without Metron to one with Metron.  The deployment was on
> AWS more to ensure we had enough memory to actually run the various
> services (the Docker cluster was having issues before we'd dumped enough
> extra services).
>
> Someone with a little more experience could probably chime in here, but
> the current AWS install actually does use the small_cluster configuration
> if you look at the defaults.yml under amazon-ec2 in metron-deployment
> package.  The mpack setup is independent of the Ansible stuff for right
> now. How close together those get and live (especially because there
> definitely is some overlap) is definitely a more involved discussion.
>
> Thanks,
> Justin
>
>
> On Wed, Sep 21, 2016 at 9:55 AM, Otto Fowler 
> wrote:
>
>> Hi Justin,
>>
>> Are you testing this against the small_cluster configuration?  With the
>> full install ( install ambari etc ) as well as the AWS install?
>> The AWS install seems like it’s own path, and is essentially different
>> from small_cluster.
>>
>> I myself am interested in the whole boat deployment - where I’m providing
>> centos nodes with only os/ssh/host setups to be totally deployed.
>>
>> On September 21, 2016 at 09:41:05, Justin Leet (justinjl...@gmail.com)
>> wrote:
>>
>> Hi all,
>>
>> I opened up a PR at  https://github.com/apache/i
>> ncubator-metron/pull/266 for everyone to take a look at and comment on.
>> For reference, the original JIRA is https://issues.apache.org/j
>> ira/browse/METRON-427
>>
>> It pretty much covers the MVP that Casey outlined and should give a
>> pretty good starting point for everyone to build on.
>>
>> There's more details on the ticket (and in the README in the code), but
>> I'll try to give the abbreviated version.
>>
>> The PR builds an mpack that sets up Kafka topics, a MySQL instance with
>> GeoIP data loaded, the Storm topologies (parsers, enrichment, and
>> indexing), and output to Elasticsearch and HDFS.  It also exposes
>> management and a lot of configuration through Ambari.  The sensors are NOT
>> managed by Ambari.  It includes some testing instructions for trying things
>> out.
>>
>> Additionally, this does not replace the current Ansible infrastructure.
>> There's definitely good discussion to be had around what interaction these
>> two approaches have.
>>
>> It also includes a set of limitations / caveats that we'll want to build
>> on as we expand out of the MVP.  I'll include them here so that everyone
>> has a good idea of what where the MVP ends (as of the PR as it stands right
>> now) and where people can contribute ideas or code if they have an interest.
>>

Re: [DISCUSS] Ambari Integration

[Otto Fowler]

Thanks Justin,

So this should just replace what is currently happening if you do the full
deployment, but you have not tested it as such?
I think the difference in the ASW deployment that I saw was how it set the
nodes to roles through the script.  Sorry if I overstated it.


On September 21, 2016 at 13:45:14, Justin Leet (justinjl...@gmail.com)
wrote:

Hi Otto,

Couple things to dig into a bit.  Let me know if I stray off what your
question is, but I think this should give you the answer.

For the mpack, it's just taking a cluster without Metron and turning it
into a cluster running Metron (regardless of the cluster itself was
provisioned).  I wasn't clear about it in my last message, but testing on
AWS wasn't really about making sure small_cluster configuration was
compatible with the mpack changes.  It was about making sure that we could
go from a cluster without Metron to one with Metron.  The deployment was on
AWS more to ensure we had enough memory to actually run the various
services (the Docker cluster was having issues before we'd dumped enough
extra services).

Someone with a little more experience could probably chime in here, but the
current AWS install actually does use the small_cluster configuration if
you look at the defaults.yml under amazon-ec2 in metron-deployment
package.  The mpack setup is independent of the Ansible stuff for right
now. How close together those get and live (especially because there
definitely is some overlap) is definitely a more involved discussion.

Thanks,
Justin


On Wed, Sep 21, 2016 at 9:55 AM, Otto Fowler 
wrote:

> Hi Justin,
>
> Are you testing this against the small_cluster configuration?  With the
> full install ( install ambari etc ) as well as the AWS install?
> The AWS install seems like it’s own path, and is essentially different
> from small_cluster.
>
> I myself am interested in the whole boat deployment - where I’m providing
> centos nodes with only os/ssh/host setups to be totally deployed.
>
> On September 21, 2016 at 09:41:05, Justin Leet (justinjl...@gmail.com)
> wrote:
>
> Hi all,
>
> I opened up a PR at  https://github.com/apache/
> incubator-metron/pull/266 for everyone to take a look at and comment on.
> For reference, the original JIRA is https://issues.apache.org/
> jira/browse/METRON-427
>
> It pretty much covers the MVP that Casey outlined and should give a pretty
> good starting point for everyone to build on.
>
> There's more details on the ticket (and in the README in the code), but
> I'll try to give the abbreviated version.
>
> The PR builds an mpack that sets up Kafka topics, a MySQL instance with
> GeoIP data loaded, the Storm topologies (parsers, enrichment, and
> indexing), and output to Elasticsearch and HDFS.  It also exposes
> management and a lot of configuration through Ambari.  The sensors are NOT
> managed by Ambari.  It includes some testing instructions for trying things
> out.
>
> Additionally, this does not replace the current Ansible infrastructure.
> There's definitely good discussion to be had around what interaction these
> two approaches have.
>
> It also includes a set of limitations / caveats that we'll want to build
> on as we expand out of the MVP.  I'll include them here so that everyone
> has a good idea of what where the MVP ends (as of the PR as it stands right
> now) and where people can contribute ideas or code if they have an interest.
>
>- MySQL install should be optional (and allow for using an existing
>instance).
>- MySQL should not be installed on a node already running a MySQL
>instance (e.g. an Ambari Server using MySQL as its database).
>- There is currently no hosting for RPMs remotely. They will have to
>be built locally.
>- Colocation of appropriate services should be enforced by Ambari. See
>'Installing Management Pack' section in the README for more details.
>- Storm's topology.classpath is not updated with the Metron service
>install and needs to be updated separately.
>- Several configuration parameters used when installing the Metron
>service could (and should) be grabbed from Ambari. Install will require
>them to be manually entered.
>- Need to handle upgrading Metron
>
>
> Thanks,
> Justin
>
> On Fri, Sep 16, 2016 at 11:32 AM, Justin Leet 
> wrote:
>
>> I went ahead and created a Jira ticket mirroring Casey's discussion of
>> the MVP.  Feel free to add anything of interest there, too.
>>
>> https://issues.apache.org/jira/browse/METRON-427
>>
>>
>> Justin
>>
>>
>> On Fri, Sep 16, 2016 at 9:39 AM, Justin Leet 
>> wrote:
>>
>>
>>> -- Forwarded message --
>>> From: zeo...@gmail.com 
>>> Date: Thu, Sep 15, 2016 at 9:02 PM
>>> Subject: Re: [DISCUSS] Ambari Integration
>>> To: u...@metron.incubator.apache.org
>>> Cc: dev@metron.incubator.apache.org
>>>
>>>
>>> Of course I would still need a full list of the 

[GitHub] incubator-metron issue #116: Metron 146 topology workers

[jjonez]

Github user jjonez commented on the issue:

https://github.com/apache/incubator-metron/pull/116
  
Sorry. I'll take care of it. 

> On Sep 21, 2016, at 1:24 PM, Nick Allen  wrote:
> 
> I am unable to. You should be able to close it though. Thanks.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
> 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Ambari Integration

[Justin Leet]

Hi Otto,

Couple things to dig into a bit.  Let me know if I stray off what your
question is, but I think this should give you the answer.

For the mpack, it's just taking a cluster without Metron and turning it
into a cluster running Metron (regardless of the cluster itself was
provisioned).  I wasn't clear about it in my last message, but testing on
AWS wasn't really about making sure small_cluster configuration was
compatible with the mpack changes.  It was about making sure that we could
go from a cluster without Metron to one with Metron.  The deployment was on
AWS more to ensure we had enough memory to actually run the various
services (the Docker cluster was having issues before we'd dumped enough
extra services).

Someone with a little more experience could probably chime in here, but the
current AWS install actually does use the small_cluster configuration if
you look at the defaults.yml under amazon-ec2 in metron-deployment
package.  The mpack setup is independent of the Ansible stuff for right
now. How close together those get and live (especially because there
definitely is some overlap) is definitely a more involved discussion.

Thanks,
Justin


On Wed, Sep 21, 2016 at 9:55 AM, Otto Fowler 
wrote:

> Hi Justin,
>
> Are you testing this against the small_cluster configuration?  With the
> full install ( install ambari etc ) as well as the AWS install?
> The AWS install seems like it’s own path, and is essentially different
> from small_cluster.
>
> I myself am interested in the whole boat deployment - where I’m providing
> centos nodes with only os/ssh/host setups to be totally deployed.
>
> On September 21, 2016 at 09:41:05, Justin Leet (justinjl...@gmail.com)
> wrote:
>
> Hi all,
>
> I opened up a PR at  https://github.com/apache/
> incubator-metron/pull/266 for everyone to take a look at and comment on.
> For reference, the original JIRA is https://issues.apache.org/
> jira/browse/METRON-427
>
> It pretty much covers the MVP that Casey outlined and should give a pretty
> good starting point for everyone to build on.
>
> There's more details on the ticket (and in the README in the code), but
> I'll try to give the abbreviated version.
>
> The PR builds an mpack that sets up Kafka topics, a MySQL instance with
> GeoIP data loaded, the Storm topologies (parsers, enrichment, and
> indexing), and output to Elasticsearch and HDFS.  It also exposes
> management and a lot of configuration through Ambari.  The sensors are NOT
> managed by Ambari.  It includes some testing instructions for trying things
> out.
>
> Additionally, this does not replace the current Ansible infrastructure.
> There's definitely good discussion to be had around what interaction these
> two approaches have.
>
> It also includes a set of limitations / caveats that we'll want to build
> on as we expand out of the MVP.  I'll include them here so that everyone
> has a good idea of what where the MVP ends (as of the PR as it stands right
> now) and where people can contribute ideas or code if they have an interest.
>
>- MySQL install should be optional (and allow for using an existing
>instance).
>- MySQL should not be installed on a node already running a MySQL
>instance (e.g. an Ambari Server using MySQL as its database).
>- There is currently no hosting for RPMs remotely. They will have to
>be built locally.
>- Colocation of appropriate services should be enforced by Ambari. See
>'Installing Management Pack' section in the README for more details.
>- Storm's topology.classpath is not updated with the Metron service
>install and needs to be updated separately.
>- Several configuration parameters used when installing the Metron
>service could (and should) be grabbed from Ambari. Install will require
>them to be manually entered.
>- Need to handle upgrading Metron
>
>
> Thanks,
> Justin
>
> On Fri, Sep 16, 2016 at 11:32 AM, Justin Leet 
> wrote:
>
>> I went ahead and created a Jira ticket mirroring Casey's discussion of
>> the MVP.  Feel free to add anything of interest there, too.
>>
>> https://issues.apache.org/jira/browse/METRON-427
>>
>>
>> Justin
>>
>>
>> On Fri, Sep 16, 2016 at 9:39 AM, Justin Leet 
>> wrote:
>>
>>
>>> -- Forwarded message --
>>> From: zeo...@gmail.com 
>>> Date: Thu, Sep 15, 2016 at 9:02 PM
>>> Subject: Re: [DISCUSS] Ambari Integration
>>> To: u...@metron.incubator.apache.org
>>> Cc: dev@metron.incubator.apache.org
>>>
>>>
>>> Of course I would still need a full list of the repos, and submit proxy
>>> rules for the Ambari box, but happy to hear it will alleviate the need
>>> for
>>> making the scripts use proxies on the cluster nodes.
>>>
>>> Jon
>>>
>>> On Thu, Sep 15, 2016, 19:34 Nick Allen  wrote:
>>>
>>> > Jon - Installing Metron on an isolated network becomes much easier with
>>> > Ambari.  You would 

[GitHub] incubator-metron issue #116: Metron 146 topology workers

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/116
  
I am unable to.  You should be able to close it though.  Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #257: METRON-426: Stellar does not support scientific...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/257
  
@justinleet I think everything you mentioned has been addressed.  Are you 
good with this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #261: METRON-434: JSON Parser

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/261
  
Casey, would you feel about adding in recursion, such that nested maps will 
be unfolded?
I have that working.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #265: METRON-438: Back the Stellar REPL with a...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/265#discussion_r79862290
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/StellarShell.java
 ---
@@ -244,6 +304,55 @@ private void write(String out) {
   }
 
   private void writeLine(String out) {
-System.out.println(out);
+console.getShell().out().println(out);
+  }
+
+  @Override
+  public int execute(ConsoleOperation output) throws InterruptedException {
+String expression = output.getBuffer().trim();
+if(StringUtils.isNotBlank(expression)) {
+  if(isMagic(expression)) {
+handleMagic( expression);
+
+  } else if(isDoc(expression)) {
+handleDoc(expression);
+
+  } else if (expression.equals("quit")) {
+try {
--- End diff --

Does this prevent the user from referring to a variable called `quit`?  If 
so, maybe `%quit`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #265: METRON-438: Back the Stellar REPL with a readli...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/265
  
@nickwallen I think I can do something about that, yep.  Let me noodle on 
just the right way to do it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #265: METRON-438: Back the Stellar REPL with a readli...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/265
  
Love it.  

One other thing I was noticing.  Not a deal breaker, but want to get your 
feedback on it.  If I auto-complete a function, it adds a space after the 
function name.  If I type "WEEK_OF_YE" then tab, it auto-completes to 
"WEEK_OF_YEAR " with a trailing space.  

If I then immediately hit ENTER, Stellar executes it, interprets it as a 
variable name, sees no value for the variable and does nothing.  Would it be 
possible to auto-complete a function with a trailing '(' so that I am forced to 
provide arguments?  Or maybe there is a better way to handle this situation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #268: METRON-421 Make Stellar Profiler Client ...

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/incubator-metron/pull/268

METRON-421 Make Stellar Profiler Client API Accessible in Parser and 
Enrichment Topologies

[METRON-421](https://issues.apache.org/jira/browse/METRON-421)

These changes were required to make the Stellar Profiler Client API 
accessible from the Parser and Enrichment topologies.

* Pushing the metron-profiler-client archive to the Metron host to make the 
library accessible to the Stellar Shell.
* Added the metron-profiler-client as a dependency to the Parser and 
Enrichment topologies so that the Uber jar contains the Profiler Client API.
* Fixed an issue with potentially unsafe String to Bytes conversion that 
may cause issues on different platforms.
* Added a README for the metron-profiler-client to describe its usage.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/incubator-metron METRON-421

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/268.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #268


commit 1ef0bf6670b3054eb0561061dc3f05fc56fa3123
Author: Nick Allen 
Date:   2016-09-19T21:23:44Z

METRON-421 Added profiler-client dependency to Enrichment and Parser 
topology so that Profile API can be used

commit 1a902c839007a8ddfcc26bf608a91c5f0d921cea
Author: Nick Allen 
Date:   2016-09-19T21:25:28Z

METRON-421 Need to push profiler-client archive to the deployed host

commit 11416ce40fad361fd8db52635f0952fd52fe25f1
Author: Nick Allen 
Date:   2016-09-20T21:47:34Z

METRON-421 Row key builder using unsafe string to bytes conversion

commit 8c46cd9be14c324f03f1b08f464ef727a52d66cb
Author: Nick Allen 
Date:   2016-09-21T14:22:54Z

METRON-421 Added README for the profiler client




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #267: METRON-445: Fix typos in metron-deployment role...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/267
  
Isn't that the British spelling? 

👍 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Ambari Integration

[Otto Fowler]

Hi Justin,

Are you testing this against the small_cluster configuration?  With the
full install ( install ambari etc ) as well as the AWS install?
The AWS install seems like it’s own path, and is essentially different from
small_cluster.

I myself am interested in the whole boat deployment - where I’m providing
centos nodes with only os/ssh/host setups to be totally deployed.

On September 21, 2016 at 09:41:05, Justin Leet (justinjl...@gmail.com)
wrote:

Hi all,

I opened up a PR at  
https://github.com/apache/incubator-metron/pull/266 for everyone to take a
look at and comment on.  For reference, the original JIRA is
https://issues.apache.org/jira/browse/METRON-427

It pretty much covers the MVP that Casey outlined and should give a pretty
good starting point for everyone to build on.

There's more details on the ticket (and in the README in the code), but
I'll try to give the abbreviated version.

The PR builds an mpack that sets up Kafka topics, a MySQL instance with
GeoIP data loaded, the Storm topologies (parsers, enrichment, and
indexing), and output to Elasticsearch and HDFS.  It also exposes
management and a lot of configuration through Ambari.  The sensors are NOT
managed by Ambari.  It includes some testing instructions for trying things
out.

Additionally, this does not replace the current Ansible infrastructure.
There's definitely good discussion to be had around what interaction these
two approaches have.

It also includes a set of limitations / caveats that we'll want to build on
as we expand out of the MVP.  I'll include them here so that everyone has a
good idea of what where the MVP ends (as of the PR as it stands right now)
and where people can contribute ideas or code if they have an interest.

   - MySQL install should be optional (and allow for using an existing
   instance).
   - MySQL should not be installed on a node already running a MySQL
   instance (e.g. an Ambari Server using MySQL as its database).
   - There is currently no hosting for RPMs remotely. They will have to be
   built locally.
   - Colocation of appropriate services should be enforced by Ambari. See
   'Installing Management Pack' section in the README for more details.
   - Storm's topology.classpath is not updated with the Metron service
   install and needs to be updated separately.
   - Several configuration parameters used when installing the Metron
   service could (and should) be grabbed from Ambari. Install will require
   them to be manually entered.
   - Need to handle upgrading Metron


Thanks,
Justin

On Fri, Sep 16, 2016 at 11:32 AM, Justin Leet  wrote:

> I went ahead and created a Jira ticket mirroring Casey's discussion of the
> MVP.  Feel free to add anything of interest there, too.
>
> https://issues.apache.org/jira/browse/METRON-427
>
>
> Justin
>
>
> On Fri, Sep 16, 2016 at 9:39 AM, Justin Leet 
> wrote:
>
>
>> -- Forwarded message --
>> From: zeo...@gmail.com 
>> Date: Thu, Sep 15, 2016 at 9:02 PM
>> Subject: Re: [DISCUSS] Ambari Integration
>> To: u...@metron.incubator.apache.org
>> Cc: dev@metron.incubator.apache.org
>>
>>
>> Of course I would still need a full list of the repos, and submit proxy
>> rules for the Ambari box, but happy to hear it will alleviate the need for
>> making the scripts use proxies on the cluster nodes.
>>
>> Jon
>>
>> On Thu, Sep 15, 2016, 19:34 Nick Allen  wrote:
>>
>> > Jon - Installing Metron on an isolated network becomes much easier with
>> > Ambari.  You would just mirror the required RPM repositories.  You can
>> then
>> > point Ambari to where your repo lives via the installation wizard.  I've
>> > done quite a few installs via Ambari on an isolated network and it
>> worked
>> > quite well.
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Sep 15, 2016 at 6:50 PM, zeo...@gmail.com 
>> > wrote:
>> >
>> >> First of all - very much looking forward to this approach.  I'm not
>> very
>> >> familiar with management packs, but I did read some of the
>> documentation in
>> >> the link you sent.
>> >>
>> >> Not sure if this is already included in a "minimum viable product," but
>> >> at some point I think there needs to be a method of specifying proxies
>> >> and/or internal package repos.  I recently did a Metron 0.2.0 install
>> >> behind a proxy (hence METRON-409
>> >> ) and it look me a
>> >> semi-lengthy amount of time to (1) find all of the destinations I
>> needed to
>> >> request openings for in the proxy, and (2) modify the ambari scripts to
>> >> appropriately use my proxies in the correct way.
>> >>
>> >> I also have a bit of a concern with upgrades and customizations in
>> >> general (Not just how it would work with mpacks).  I have not done any
>> of
>> >> this to date, but I have rebuilt and redeployed a couple of times and I
>> >> needed to modify some of the metron code itself before 

Re: [DISCUSS] Ambari Integration

[Justin Leet]

Hi all,

I opened up a PR at  
https://github.com/apache/incubator-metron/pull/266 for everyone to take a
look at and comment on.  For reference, the original JIRA is
https://issues.apache.org/jira/browse/METRON-427

It pretty much covers the MVP that Casey outlined and should give a pretty
good starting point for everyone to build on.

There's more details on the ticket (and in the README in the code), but
I'll try to give the abbreviated version.

The PR builds an mpack that sets up Kafka topics, a MySQL instance with
GeoIP data loaded, the Storm topologies (parsers, enrichment, and
indexing), and output to Elasticsearch and HDFS.  It also exposes
management and a lot of configuration through Ambari.  The sensors are NOT
managed by Ambari.  It includes some testing instructions for trying things
out.

Additionally, this does not replace the current Ansible infrastructure.
There's definitely good discussion to be had around what interaction these
two approaches have.

It also includes a set of limitations / caveats that we'll want to build on
as we expand out of the MVP.  I'll include them here so that everyone has a
good idea of what where the MVP ends (as of the PR as it stands right now)
and where people can contribute ideas or code if they have an interest.

   - MySQL install should be optional (and allow for using an existing
   instance).
   - MySQL should not be installed on a node already running a MySQL
   instance (e.g. an Ambari Server using MySQL as its database).
   - There is currently no hosting for RPMs remotely. They will have to be
   built locally.
   - Colocation of appropriate services should be enforced by Ambari. See
   'Installing Management Pack' section in the README for more details.
   - Storm's topology.classpath is not updated with the Metron service
   install and needs to be updated separately.
   - Several configuration parameters used when installing the Metron
   service could (and should) be grabbed from Ambari. Install will require
   them to be manually entered.
   - Need to handle upgrading Metron


Thanks,
Justin

On Fri, Sep 16, 2016 at 11:32 AM, Justin Leet  wrote:

> I went ahead and created a Jira ticket mirroring Casey's discussion of the
> MVP.  Feel free to add anything of interest there, too.
>
> https://issues.apache.org/jira/browse/METRON-427
>
>
> Justin
>
>
> On Fri, Sep 16, 2016 at 9:39 AM, Justin Leet 
> wrote:
>
>
>> -- Forwarded message --
>> From: zeo...@gmail.com 
>> Date: Thu, Sep 15, 2016 at 9:02 PM
>> Subject: Re: [DISCUSS] Ambari Integration
>> To: u...@metron.incubator.apache.org
>> Cc: dev@metron.incubator.apache.org
>>
>>
>> Of course I would still need a full list of the repos, and submit proxy
>> rules for the Ambari box, but happy to hear it will alleviate the need for
>> making the scripts use proxies on the cluster nodes.
>>
>> Jon
>>
>> On Thu, Sep 15, 2016, 19:34 Nick Allen  wrote:
>>
>> > Jon - Installing Metron on an isolated network becomes much easier with
>> > Ambari.  You would just mirror the required RPM repositories.  You can
>> then
>> > point Ambari to where your repo lives via the installation wizard.  I've
>> > done quite a few installs via Ambari on an isolated network and it
>> worked
>> > quite well.
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Sep 15, 2016 at 6:50 PM, zeo...@gmail.com 
>> > wrote:
>> >
>> >> First of all - very much looking forward to this approach.  I'm not
>> very
>> >> familiar with management packs, but I did read some of the
>> documentation in
>> >> the link you sent.
>> >>
>> >> Not sure if this is already included in a "minimum viable product," but
>> >> at some point I think there needs to be a method of specifying proxies
>> >> and/or internal package repos.  I recently did a Metron 0.2.0 install
>> >> behind a proxy (hence METRON-409
>> >> ) and it look me a
>> >> semi-lengthy amount of time to (1) find all of the destinations I
>> needed to
>> >> request openings for in the proxy, and (2) modify the ambari scripts to
>> >> appropriately use my proxies in the correct way.
>> >>
>> >> I also have a bit of a concern with upgrades and customizations in
>> >> general (Not just how it would work with mpacks).  I have not done any
>> of
>> >> this to date, but I have rebuilt and redeployed a couple of times and I
>> >> needed to modify some of the metron code itself before build/deploy
>> >> (because of my concern with it getting overwritten on upgrade if I
>> just did
>> >> it directly on the cluster).  I would like to see a method of putting
>> in
>> >> install-specific files that modify or overwrite parts of the core
>> metron
>> >> stack, like changes to znodes, parsers, etc.
>> >>
>> >> Regarding not managing sensors with Ambari, I agree.  I run a large bro
>> >> cluster and it is maintained via Puppet and various other 

[GitHub] incubator-metron pull request #267: METRON-445: Fix typos in metron-deployme...

[JonZeolla]

GitHub user JonZeolla opened a pull request:

https://github.com/apache/incubator-metron/pull/267

METRON-445: Fix typos in metron-deployment roles

Apply s/passowrd/password/ to the main.yml files under 
metron-deployment/roles/{mysql_client,metron_streaming}/tasks/.  

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JonZeolla/incubator-metron master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/267.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #267


commit d87c6daa0c15dbd53c26067c659b4c11141a9e36
Author: Jon Zeolla 
Date:   2016-09-21T13:37:35Z

METRON-445: Fix typos in metron-deployment roles




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #266: METRON-427 Create Ambari Management Pack...

[justinleet]

GitHub user justinleet opened a pull request:

https://github.com/apache/incubator-metron/pull/266

METRON-427 Create Ambari Management Pack for Metron Installation

This is an initial cut at an MVP for the Ambari Management Pack.  Most of 
the ground work was completed in other Jiras (detailed in METRON-427 
description).  This takes that work, adds the enrichment portion similiar to 
indexing, elasticsearch, etc., and bundles it in an mpack.

This includes end to end from creating appropriate Kafka topics, preloading 
a MySQL database with GeoIP data for enrichment, and making it available to an 
Elasticsearch and Kibana instance. ES is optional, and can be redirected to an 
outside instance.

The directory structuring changed to reflect the mpack MVP, and accordingly 
a lot of files got moved/renamed.  For most of the ground work, there are 
various fixes and tweaks for consistency to each other and to fix bugs found 
during testing.

Detailed instructions are found in the metron-deployment README.md in the 
'Ambari Management Pack' section.  Essentially these steps are:
- Build the RPMs per the previously existing instructions (make sure to run 
the main build first).
- Build the mpack using the metron-deployment POM. From metron-deployment 
just run `mvn clean package -DskipTests`.  This will produce an mpack tar.gz, 
which is what will be used in the installation to the Ambari server.
- Place the RPMs in /localrepo on the nodes.  We don't have remote RPMs 
yet, so this is a necessary evil that will go away.
- Place the mpack file on the ambari-server node.
- Install the mpack with `ambari-server` install-mpack 
--mpack= --verbose`
- Update Storm's configuration in Ambari for topology.classpath to 
/etc/hbase/conf:/etc/hadoop/conf.  Ideally this will be automated after this 
MVP.
- Run through the normal Ambari install process.  There are some caveats 
about this (colocation of Metron services isn't enforced, and Storm + Kafka + 
HBase client need to be available).  More details are in the README, but just 
make sure Storm + Kafka + HBase client are available and that you aren't 
installing to a node with MySQL already installed.  The MySQL issue is most 
likely on a single node install
- Use the action to install Elasticsearch templates where you can start / 
stop services.

This was tested on an AWS cluster, and on a set of Docker instances where a 
lot of components (e.g. MapReduce) disabled to avoid memory issues.

Steps used on Docker were (the server and agent images already have most of 
the unnecessary components disabled.  Make sure to change jleet to your user 
home:
`docker-machine rm default
docker-machine create --driver virtualbox --virtualbox-disk-size "51200" 
--virtualbox-memory "8192" default

docker-machine start default
eval $(docker-machine env default)
git clone https://github.com/sequenceiq/docker-ambari
cd docker-ambari
. ambari-functions
export AMBARI_SERVER_IMAGE=dlyle65535/ambari-server:2.4.0.1-jdk8
export AMBARI_AGENT_IMAGE=dlyle65535/ambari-agent:2.4.0.1-jdk8

export DOCKER_OPTS="-v /Users/jleet:/home/host -v 
/Users/jleet/Documents/workspace/incubator-metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch:/localrepo
 -v 
/Users/jleet/Documents/workspace/incubator-metron/metron-deployment/packaging/ambari/metron-mpack/target/:/mpack"

(mac only) sudo route add -net 172.17.0.0/16 $(docker-machine ip default)

amb-start-cluster 5
amb-shell
blueprint add --url 
https://raw.githubusercontent.com/dlyle65535/ambari-rest-client/docker-metron-test/src/main/resources/blueprints/docker-metron
cluster build --blueprint docker-metron
cluster assign --host amb1.service.consul --hostGroup master_1
cluster assign --host amb2.service.consul --hostGroup master_2
cluster assign --host amb3.service.consul --hostGroup master_3
cluster assign --host amb4.service.consul --hostGroup master_4
cluster create



docker exec -it amb-server bash
ambari-server install-mpack 
--mpack=/mpack/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz --verbose
ambari-server restart
http://amb-server.service.consul:8080 - install Metron service on 
amb4.service.consul
es_url: can be invalid for test purpose otherwise, use authentic 
   One can docker it - docker run -d -p 9200:9200 -p 9300:9300 
elasticsearch -Des.cluster.name=metron
Storm UI Server: amb1.service.consul:8744
`




Limitations of the MVP (also detailed in README)
- MySQL install should be optional (and allow for using an existing 
instance).
- MySQL should not be installed on a node already running a MySQL instance 
(e.g. an Ambari Server using MySQL as its database).
- There is currently no hosting for RPMs remotely.  They will have to be 
built 

[GitHub] incubator-metron issue #261: METRON-434: JSON Parser

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/261
  
@ottobackwards Regarding your other question, we mandated that every 
message capture the original message and a timestamp as required fields.  If 
the message doesn't have that, then it fails validation and an error is thrown 
for that message.

Certainly after it's parsed, you can apply a stellar transformation and 
replace the timestamp field with another one using Stellar.

There is currently a bug where we apply stellar transformations AFTER that 
validation happens, so we don't have the ability to specify a timestamp via 
stellar if it is not there already.  If we had that fixed, then we could apply 
the timestamp to the message in stellar, for sure.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Hello Metron

[Otto Fowler]

Hi everyone,

My name is Otto Fowler, and I work at Leidos Cyber ( formerly Lockheed
Martin IS ).  I am very impressed with the Metron project and the work
everyone has been doing and I’ve really enjoyed working with Metron so far
in my evaluation.  I look forward to participating in this new but growing
community!