[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...

Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/455 I've tested it to completion on both Centos 6 and 7. It works and generates identical src and site file trees. I also tested the trap on SIGINT, which worked on all three platforms.

[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...

Github user JonZeolla commented on the issue: https://github.com/apache/incubator-metron/pull/455 Spun it up on CentOS 6.8 and macOX 10.12.3, and did a quick code review. Spot tested the traps as well. +1 (non-binding). The only thing I noticed is I don't think

[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...

Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/455 I have pushed fixes for all of the above. To fix item 3 I had to change history, so you'll need to "pull -f" to update an existing image. I tested code fragments and

Re: [DISCUSS][PROPOSAL] Side Loading and Installation of telemetry sources [METRON-258]

Outstanding write-up, Otto! As Casey said, don’t expect this to be a coherent response, but some possibly useful thoughts: 1. It’s clear that because parsers, enrichers, and indexers are all specialized per sensor, that “adding a new sensor” is necessarily a complex operation. You’ve thrown

Re: [DISCUSS] Metron Management UI

Well, I think that a UI that does what you show is a huge step forward and I definitely wouldn't want to stop it from getting into the code because of any of the additional features that we are talking about here. That said, I really would like to see the Kafka topic creation as a part of the UI

Re: [DISCUSS] Management of Elastic and other index schemas

I think this is a good direction to move things toward - moving indexing templates to be packaged with parsers (using multiple tiered options) that are then merged with the possible enrich fields before getting added to the indexing technology in use. Now, to read the proposal thread... Jon On

[GitHub] incubator-metron issue #453: METRON-694: Index Errors from Topologies

Github user merrimanr commented on the issue: https://github.com/apache/incubator-metron/pull/453 I think removing the error indexing topology is fine as long as we're careful to avoid error messages getting stuck in a loop in the case of a failed index write. I agree that it will

Re: [DISCUSS] Management of Elastic and other index schemas

I’d broadly agree with that tiered approach. The version where the parser emits a generic schema, and enrichments contribute generic schema chunks to that which get combined into an indexer specific template generated at the end of the flow, so yes, pretty much inline with your proposal. (I

[GitHub] incubator-metron pull request #457: METRON-724 add [IN] to the documented ke...

Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/457 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

Re: [DISCUSS][PROPOSAL] Side Loading and Installation of telemetry sources [METRON-258]

RE: * One Module - yes, I think grouping for the base parsers is good, I just don’t want them to stay in -common, it should ‘live’ in the metron lib. I think a grouped set of the primitive parsers is correct, still it’s own. * ES Templates - they don’t *have* to be there, but if they are they

[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...

Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/455 Hi @JonZeolla , @ottobackwards , @cestella , @mmiklavc , Thanks to all of you for looking at this. Taking issues in order: 1. Absolutely right, no reason to have site.xml.bak

Re: [DISCUSS][PROPOSAL] Side Loading and Installation of telemetry sources [METRON-258]

Ok, This is a long one, so don't expect a coherent response just yet, but I will give some initial impressions: - I strongly agree with the premise of this idea. Making Metron extensible is and should be among the top of our priorities and at the moment, it's painful to develop a new

[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/455 ok - i'm not building this branch, just what i've seen from other builds --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

Re: [DISCUSS] Management of Elastic and other index schemas

We already make them do this now, or they get the defaults. So this is no different. Having parsers emit names and types etc, that would be another step - or it could be the ‘generic schema’ as implemented actually. A tiered approach - from * you give nothing with the parser - you get whatever

[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...

Github user mmiklavc commented on the issue: https://github.com/apache/incubator-metron/pull/455 Hey @ottobackwards site.xml is now in .gitignore with @mattf-horton's latest changes. --- If your project is set up for it, you can reply to this email and have your reply appear on

Re: [DISCUSS] Management of Elastic and other index schemas

I like that, to an extent… Forcing the provision of explicit schema might be a bit of a load for parser development. I’m assuming that custom parsers would be pushed towards the same packaging approach. Would it make sense to require the parser to emit field names and types expected, and then

[GitHub] incubator-metron issue #453: METRON-694: Index Errors from Topologies

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/453 Ok, catching up on this: Regarding invalid vs errors, I'm ok with the notion of treating them the same, but it's different from how we've done this in the past. Prior to this,

[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/455 re: "site.xml is now a fully auto-generated file" that would be why it is showing in git as unstaged but modified? Is there a way we can stop that from happening? If it is

[GitHub] incubator-metron pull request #457: METRON-724 add [IN] to the documented ke...

Github user cestella commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/457#discussion_r101838232 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java --- @@ -288,6 +289,26 @@ public void

[GitHub] incubator-metron issue #457: METRON-724 add [IN] to the documented keywords ...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/457 +1 by inspection --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

Re: Cannot close JIRAs I didn't originally request

For what it’s worth, I agree with Jon and Billie that “somebody” (ie, a PMC member) should post an INFRA ticket requesting “Contributors” be given the “Transition Issues” permission in the METRON project in Jira. --Matt On 2/17/17, 9:37 AM, "zeo...@gmail.com" wrote:

[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/455 Either as @JonZeolla says or you could also do `basename $md .md`. I do think running on centos is important as eventually I'd like to get this running as part of travis, so I'm willing

Re: [DISCUSS] Management of Elastic and other index schemas

I think we can get there from my proposal. A source may package: * explicit schemas ( ES, SOLR, FOO ) * a generic to be invented schema for a to be invented pluggable indexing component :) and we’ll be able to handle it. On February 17, 2017 at 14:39:07, Kyle Richardson

Re: [DISCUSS] Management of Elastic and other index schemas

I just sent the discuss/propose email out Simon. I was thinking to start just packaging the equiv. of what we have now - es template and logrotate scripts would be the way to go. If we have solr, then those would go in there too. I don’t want to split the thread - but I have some ui workflow

[DISCUSS][PROPOSAL] Side Loading and Installation of telemetry sources [METRON-258]

The ability for implementors and developers building on the project to ‘side load’, that is to build, maintain, and install, telemetry sources into the system without having to actually develop within METRON itself is very important. If done properly it gives developers and easier and more

[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...

Github user JonZeolla commented on the issue: https://github.com/apache/incubator-metron/pull/455 I'm a -1 (non-binding) on this for now, primarily because the bash fails to run on CentOS 6.8 (where I tested it). The error was `basename: invalid option -- 's'`. I think I

Re: [DISCUSS] Management of Elastic and other index schemas

I personally like the idea of a typed schema per parser that we could translate to multiple targets. This would allow us a lot more modularity and extensibility in indexing down the road. -Kyle On Fri, Feb 17, 2017 at 1:59 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > That

Re: [DISCUSS] Management of Elastic and other index schemas

That sounds like a great idea Otto. Do you have any early design on that we can look at. Also, rather than just elastic templates do you think we should have some sort of typed schema we could translate to multiple targets (solr, elastic, ur... other...) or are you thinking of packaging

[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/455 Ok, yep, I'm +1 on this. Also, I agree with the reasoning around not having a bak file. Especially compelling is the first bullet point that `site.xml` is generated now. --- If your

[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...

Github user mattf-horton commented on the issue: https://github.com/apache/incubator-metron/pull/455 Hi @JonZeolla , thanks for taking a look amidst the other demands on your time. I disagree with 'mv $fullpath/site.xml.bak $fullpath/site.xml' for the following reasons:

Re: [DISCUSS] Management of Elastic and other index schemas

Not to jump the gun, but I’m crafting a proposal about parsers and one of the things I am going to propose relates to having the ES Template for a given parser installed or packaged with the parser. We could load the template from there, edit, save and deploy etc. We can extend that concept more

Re: [DISCUSS] Management of Elastic and other index schemas

If we had a UI to switch types for existing fields in a template, would that work? What else is necessary? On 2/17/17, 10:22 AM, "Simon Elliston Ball" wrote: >A little while ago the issue of managing Elastic templates for new sensor >configs came up, and we

Re: [DISCUSS] Metron Management UI

Hi Dima and Jon, Do you think the UI: http://imgur.com/a/QAQyu? ...Is a reasonable place to start as it is?… or do we need to have the Elastic Search Templates right away? Are the templates a MUST HAVE in order to add this UI to Metron? As Simon said, I think we can file a JIRA on the

Re: [DISCUSS] Metron Management UI

Seems like the UI discussion has fallen off the inboxes. We should definitely open up a separate Jira and discussion on the best way to process elastic search template management. This feels like a non-UI question to me, since it would apply equally to any other parser config method. I’ll

[GitHub] incubator-metron issue #450: METRON-690: Create a DSL-based timestamp lookup...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/450 Testing Instructions beyond the normal smoke test (i.e. letting data flow through to the indices and checking them). ## Free Up Space on the virtual machine First,

Re: Cannot close JIRAs I didn't originally request

Thoughts? Just want to put this one to rest, one way or another. Jon On Fri, Feb 3, 2017 at 8:03 PM zeo...@gmail.com wrote: > Has anybody had a chance to look into this and decide whether a change > should be made? This specific incident is no longer an issue but I'd like >

[GitHub] incubator-metron issue #316: METRON-503: Metron REST API

Github user merrimanr commented on the issue: https://github.com/apache/incubator-metron/pull/316 The latest commit includes removing MySQL as the security credential store and replacing with H2. I also updated the documentation with instructions on configuring a production database

[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...

Github user JonZeolla commented on the issue: https://github.com/apache/incubator-metron/pull/455 I'm not going to have time to review this more than I already have. +1 (non-binding) via inspection. If the PR is still open in 1 1/2 weeks I can take another stab at a review, but

[GitHub] incubator-metron pull request #457: METRON-724 add [IN] to the documented ke...

GitHub user ottobackwards opened a pull request: https://github.com/apache/incubator-metron/pull/457 METRON-724 add [IN] to the documented keywords that require escaping The keyword 'in' must be escaped to be used in Stellar, but is not documented as such. This PR adds in the

[GitHub] incubator-metron issue #316: METRON-503: Metron REST API

Github user jjmeyer0 commented on the issue: https://github.com/apache/incubator-metron/pull/316 @merrimanr sorry for not commenting earlier. I think this is in a pretty good state. I do want to review this again from beginning to end after you finish the MySQL stuff. After that I'll

[GitHub] incubator-metron pull request #456: METRON-721 Add github pull request templ...

Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/456 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] incubator-metron issue #450: METRON-690: Create a DSL-based timestamp lookup...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/450 @james-sirota Yep, agreed. I'll modify the docs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] incubator-metron issue #456: METRON-721 Add github pull request template to ...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/456 +1 from me by inspection. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...

Github user JonZeolla commented on the issue: https://github.com/apache/incubator-metron/pull/455 Sounds good to me. I typically use declare for all variables just to be explicit and obvious with my intent, and use $() as a standard for readability and easy nesting as opposed to