[GitHub] incubator-metron pull request #536: METRON-862 Add Mattf gpg public key to K...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/536


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #536: METRON-862 Add Mattf gpg public key to K...

[mattf-horton]

GitHub user mattf-horton opened a pull request:

https://github.com/apache/incubator-metron/pull/536

METRON-862 Add Mattf gpg public key to KEYS file

## Pull Request Checklist

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

### For code changes: NA

### For documentation related changes: NA


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattf-horton/incubator-metron METRON-862

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/536.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #536


commit 193c6f15cb3aafc9052387a3a09b594c6de8cc8e
Author: mattf-horton 
Date:   2017-04-19T05:51:51Z

METRON-862 Add Mattf gpg public key to KEYS file




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #527: METRON-848: MPack's metron_theme.json is...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/527


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: MaaS + Apache Twill ?

[James Sirota]

I would stick to MaaS.  We already have service discovery and scaling built 
into MaaS so we really don't have a need for features that Twill offers.  I 
think had we known that Twill existed a few years ago we would have considered 
it, but given where we are today I don't see a reason to switch.

Thanks,
James 

18.04.2017, 12:30, "Nick Allen" :
>>   Can Twill handle long-running applications?
>
> I did see this mentioned in the slide deck for "Big Data North America
> 2016" around slide 33 or so. The deck is referenced from their home page.
> If we blindly trust the slide deck, then yes, it does support this. :)
>
> On Tue, Apr 18, 2017 at 3:12 PM, Casey Stella  wrote:
>
>>  Some things to look for that address some of the complexities within MaaS:
>>
>> - Can Twill handle long-running applications?
>> - Does Twill provide any sort of name service abstraction or somesuch
>> that lets us communicate with the app master after app deployment?
>> - Does Twill provide any mechanism for impersonation (i.e. we can run
>> the MaaS twill app as metron, but submit and run models as $user)?
>>
>>  On Tue, Apr 18, 2017 at 1:50 PM, Nick Allen  wrote:
>>
>>  > I ran across the Apache Twill [1] project recently whose goal is to
>>  reduce
>>  > the complexity of developing distributed applications that run on YARN.
>>  My
>>  > first thought is that it might offer additional capabilities and/or
>>  > simplify our current MaaS implementation.
>>  >
>>  > Here are a list of features provided by Twill that I think might be
>>  useful
>>  > for MaaS.
>>  >
>>  > - Service discovery
>>  > - Elastic scaling
>>  > - High Availability
>>  > - Placement policies - Which rack/host should the model run on?
>>  > - Security - Kerberos ticket refresh?
>>  >
>>  > Just wanted to float the thought in the community and see if anyone has
>>  > experience with Twill. I need to do some more research myself.
>>  >
>>  > [1] http://twill.apache.org/
>>  >

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Re: MaaS + Apache Twill ?

[Casey Stella]

One more thought, I definitely am not opposed to making it distributed
resource manager independent.  I think what Otto is suggesting isn't a bad
thread to pull on.  Right now, MaaS is tied to Yarn inherently and it'd be
nice to make that dependency pluggable.  This would allow us to use other
Lambda or Kubernetes or whatever for model deployment, which would be
really neat.

On Tue, Apr 18, 2017 at 3:02 PM, Casey Stella  wrote:

> Regarding model performance, I've thought about that a bit.  What I'd like
> to see MaaS be able to do is provide an API that the models can communicate
> through that will send events to kafka and provides a telemetry like any
> other.  Performance statistics, raw results for downstream analysis.  We
> have a system capable of analyzing telemetry data, it seems to me like MaaS
> should use the dogfood in which it runs.  I think we should build an API by
> which the model can communicate with kafka.
>
> One of the things that I very much like about MaaS is that it's light on
> the opinion about the language and library.  I think that building the API
> should be as simple as a log file that is monitored and the MaaS runner
> will provide that proxy to kafka.  I do think that it should be easier to
> write models, but I think that should be solved through applications that
> will turn model collateral into REST APIs if they conform to certain
> standards (i.e. PMML, Spark MLLib serialized models, etc.) and allow users
> the freedom to engage with MaaS via their own mechanism in the language of
> their choice if their situation doesn't conform to our expectations.
>
> On Tue, Apr 18, 2017 at 2:50 PM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
>> Completely agree. It would be great to get more info from your experience.
>>
>> To an extent what we have at the moment is very much about isolating the
>> implementation of a model from the deployment and discovery mechanism, and
>> to my mind we should very much keep that to enable any kind of model to
>> plugin. The other thing worth discussing here would be how we can wrap
>> around a model while maintaining the lose coupling, to provide things like
>> generalised performance metrics for the models. Any thoughts on that front
>> anyone?
>>
>>
>> > On 18 Apr 2017, at 11:45, Otto Fowler  wrote:
>> >
>> > I will have to go back to my notes.  There was a day or so when I went
>> through the code and was thinking of a couple of things, but that was a
>> while ago.
>> >
>> > Off the top of my head, I would want something factored enough, or
>> loosely coupled enough that it was not dependent on twill or maas or
>> anything else.  This
>> > would not impose implementation on the services.   This would have to
>> revolve around a discovery/registration api and a rest interface contract.
>> >
>> > Does that make sense?
>> >
>> >
>> >
>> >
>> > On April 18, 2017 at 14:34:28, Simon Elliston Ball (
>> si...@simonellistonball.com ) wrote:
>> >
>> >> Right, how about some sort of REST API? Or through Ambari? What would
>> you say was the best way to start the service, and of course to submit
>> model artefacts?
>> >>
>> >> Simon
>> >>
>> >>> On 18 Apr 2017, at 11:33, Otto Fowler > > wrote:
>> >>>
>> >>> In my mind, I didn’t want to deploy the service as a bash script or
>> wrapped in one, if I recall correctly.
>> >>>
>> >>>
>> >>>
>> >>> On April 18, 2017 at 14:27:52, Simon Elliston Ball (
>> si...@simonellistonball.com ) wrote:
>> >>>
>>  Any particular issues, or things that didn’t work Otto?
>> 
>>  Simon
>> 
>> 
>>  > On 18 Apr 2017, at 11:26, Otto Fowler > > wrote:
>>  >
>>  > I’ll try to take a look. There are a couple of things I wanted to
>> do with MaaS but could not
>>  > figure out because of a couple of limitations. I’d like to see if
>> twill offers more flexibility
>>  >
>>  >
>>  >
>>  >
>>  > On April 18, 2017 at 13:50:39, Nick Allen (n...@nickallen.org
>> ) wrote:
>>  >
>>  > I ran across the Apache Twill [1] project recently whose goal is
>> to reduce
>>  > the complexity of developing distributed applications that run on
>> YARN. My
>>  > first thought is that it might offer additional capabilities and/or
>>  > simplify our current MaaS implementation.
>>  >
>>  > Here are a list of features provided by Twill that I think might
>> be useful
>>  > for MaaS.
>>  >
>>  > - Service discovery
>>  > - Elastic scaling
>>  > - High Availability
>>  > - Placement policies - Which rack/host should the model run on?
>>  > - Security - Kerberos ticket refresh?
>>  >
>>  > Just wanted to float the thought in the community and see if
>> anyone has
>> 

Re: MaaS + Apache Twill ?

[Casey Stella]

Regarding model performance, I've thought about that a bit.  What I'd like
to see MaaS be able to do is provide an API that the models can communicate
through that will send events to kafka and provides a telemetry like any
other.  Performance statistics, raw results for downstream analysis.  We
have a system capable of analyzing telemetry data, it seems to me like MaaS
should use the dogfood in which it runs.  I think we should build an API by
which the model can communicate with kafka.

One of the things that I very much like about MaaS is that it's light on
the opinion about the language and library.  I think that building the API
should be as simple as a log file that is monitored and the MaaS runner
will provide that proxy to kafka.  I do think that it should be easier to
write models, but I think that should be solved through applications that
will turn model collateral into REST APIs if they conform to certain
standards (i.e. PMML, Spark MLLib serialized models, etc.) and allow users
the freedom to engage with MaaS via their own mechanism in the language of
their choice if their situation doesn't conform to our expectations.

On Tue, Apr 18, 2017 at 2:50 PM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> Completely agree. It would be great to get more info from your experience.
>
> To an extent what we have at the moment is very much about isolating the
> implementation of a model from the deployment and discovery mechanism, and
> to my mind we should very much keep that to enable any kind of model to
> plugin. The other thing worth discussing here would be how we can wrap
> around a model while maintaining the lose coupling, to provide things like
> generalised performance metrics for the models. Any thoughts on that front
> anyone?
>
>
> > On 18 Apr 2017, at 11:45, Otto Fowler  wrote:
> >
> > I will have to go back to my notes.  There was a day or so when I went
> through the code and was thinking of a couple of things, but that was a
> while ago.
> >
> > Off the top of my head, I would want something factored enough, or
> loosely coupled enough that it was not dependent on twill or maas or
> anything else.  This
> > would not impose implementation on the services.   This would have to
> revolve around a discovery/registration api and a rest interface contract.
> >
> > Does that make sense?
> >
> >
> >
> >
> > On April 18, 2017 at 14:34:28, Simon Elliston Ball (
> si...@simonellistonball.com ) wrote:
> >
> >> Right, how about some sort of REST API? Or through Ambari? What would
> you say was the best way to start the service, and of course to submit
> model artefacts?
> >>
> >> Simon
> >>
> >>> On 18 Apr 2017, at 11:33, Otto Fowler  > wrote:
> >>>
> >>> In my mind, I didn’t want to deploy the service as a bash script or
> wrapped in one, if I recall correctly.
> >>>
> >>>
> >>>
> >>> On April 18, 2017 at 14:27:52, Simon Elliston Ball (
> si...@simonellistonball.com ) wrote:
> >>>
>  Any particular issues, or things that didn’t work Otto?
> 
>  Simon
> 
> 
>  > On 18 Apr 2017, at 11:26, Otto Fowler  > wrote:
>  >
>  > I’ll try to take a look. There are a couple of things I wanted to
> do with MaaS but could not
>  > figure out because of a couple of limitations. I’d like to see if
> twill offers more flexibility
>  >
>  >
>  >
>  >
>  > On April 18, 2017 at 13:50:39, Nick Allen (n...@nickallen.org
> ) wrote:
>  >
>  > I ran across the Apache Twill [1] project recently whose goal is to
> reduce
>  > the complexity of developing distributed applications that run on
> YARN. My
>  > first thought is that it might offer additional capabilities and/or
>  > simplify our current MaaS implementation.
>  >
>  > Here are a list of features provided by Twill that I think might be
> useful
>  > for MaaS.
>  >
>  > - Service discovery
>  > - Elastic scaling
>  > - High Availability
>  > - Placement policies - Which rack/host should the model run on?
>  > - Security - Kerberos ticket refresh?
>  >
>  > Just wanted to float the thought in the community and see if anyone
> has
>  > experience with Twill. I need to do some more research myself.
>  >
>  > [1] http://twill.apache.org/ 
>

Re: [GitHub] incubator-metron issue #527: METRON-848: MPack's metron_theme.json is malfor...

[Otto Fowler]

+1


On April 18, 2017 at 14:39:11, dlyle65535 (g...@git.apache.org) wrote:

Github user dlyle65535 commented on the issue: 

https://github.com/apache/incubator-metron/pull/527 

If there's no objection to including this in 0.4, I'd like to do that as well. 

@mattf-horton - are you okay with my committing to master and 0.4? 



--- 
If your project is set up for it, you can reply to this email and have your 
reply appear on GitHub as well. If your project does not have this feature 
enabled and wishes so, or if the feature is enabled but not working, please 
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket 
with INFRA. 
--- 

[GitHub] incubator-metron pull request #533: METRON-856: Ansible rpm build wipes out ...

[merrimanr]

GitHub user merrimanr opened a pull request:

https://github.com/apache/incubator-metron/pull/533

METRON-856: Ansible rpm build wipes out prior binary build

## Contributor Comments
Tested this on full dev and verified the correct profile is being used.  I 
was able to follow the instructions in Kerberos-setup.md without issue.  Build 
time seemed to go down too.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron (Incubating).  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-856

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/533.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #533


commit 17e3b273bda19d5365e177c1bc8d00fbb226e378
Author: merrimanr 
Date:   2017-04-18T15:24:08Z

updated rpm build ansible task




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Metron + HDP 2.5.3 + Ambari + Centos 7 = pain

[Otto Fowler]

If this resolved your issue, can you comment on the pull request?



On April 18, 2017 at 12:37:53, Anand Subramanian (
asubraman...@hortonworks.com) wrote:

Hi Otto,

I did bump into the same issue you described - just a spinning Next button.

Applying this worked for me. Can you give it a try?
https://github.com/apache/incubator-metron/pull/527


-Anand




On 4/18/17, 8:06 PM, "Kyle Richardson"  wrote:

>I think you may be running into a bug that Matt just submitted a PR for.
>Have a read through METRON-634 (PR #532) it may resolve the Ambari / ES /
>Centos 7 issue you're seeing.
>
>-Kyle
>
>On Mon, Apr 17, 2017 at 5:35 PM, Otto Fowler 
>wrote:
>
>> These would apply for centos 7 as well then?
>>
>>
>> On April 17, 2017 at 17:06:51, JJ Meyer (jjmey...@gmail.com) wrote:
>>
>> Otto,
>>
>> I haven't tried that article, but I have used the one on our wiki, and
have
>> had success. It's based on the one you link to, but I *believe* it may
have
>> more steps:
>>
>> https://cwiki.apache.org/confluence/display/METRON/
>> Metron+with+HDP+2.5+bare-metal+install
>>
>> Thanks,
>> JJ
>>
>> On Mon, Apr 17, 2017 at 3:43 PM, Otto Fowler 
>> wrote:
>>
>> > Has anyone run these steps recently?
>> > https://community.hortonworks.com/articles/60805/deploying-
>> > a-fresh-metron-cluster-using-ambari-serv.html
>> > I am not having any luck. I’m seeing ambari hang after clicking the
next
>> > button on the setup assign slaves page.
>> > Also seeing it hang if I install service by service, once I get to
>> > metron+kibana+es.
>> >
>> > I don’t see any errors in the log files.
>> >
>>

[GitHub] incubator-metron pull request #308: Metron-498 Grok patterns are now read fr...

[merrimanr]

Github user merrimanr closed the pull request at:

https://github.com/apache/incubator-metron/pull/308


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---