Re: [DISCUSS] Deprecate Least Recently Used Pruner
I think setting TTL values for enrichments should be fine for the time being.
I am not sure what the issue was originally where we didn't feel it was
sufficient (probably performance implication on Hbase?), but I think it should
be safe reverting back to setting TTL.
James
13.08.2019, 15:44, "Casey Stella" :
> Ah, that feature. Yes, it never seemed to catch on. It actually wasn't
> from OpenSOC, but a very early feature of Metron. The use-case was that
> enrichments may go stale and removing them based on TTL was easy to do, but
> not ideal. The LeastRecentlyUsedPruner was a MR job which would allow
> enrichments to be pruned which had not been *read* in x amount of time. It
> did this by capturing bloom filters with enrichment keys used for a
> time-range and the MR job would use those bloom filters to determine which
> keys to remove.
>
> I'd be ok with it either being used or removed. It's unclear to me whether
> the use-case that hbase needs to be pruned based on usage was as valid as
> we thought. I guess that makes me +0 on the request to deprecate.
>
> On Tue, Aug 13, 2019 at 6:28 PM Nick Allen wrote:
>
>> Sure. I should have provided some more context. I can tell you what I do
>> know about it. Perhaps others can provide some more color.
>>
>> - This is functionality accessed by a user by running the script; ${
>> METRON_HOME}/bin/threatintel_bulk_prune.sh
>>
>> - If you are using access trackers with your HBase enrichments, it runs
>> as an MR job that counts the number of times each Enrichment is used.
>> I am
>> assuming that it then prunes those that are less frequently accessed.
>>
>> - It was originally created here;
>> https://github.com/apache/metron/pull/22
>>
>> On Tue, Aug 13, 2019 at 6:11 PM Otto Fowler
>> wrote:
>>
>> > Can you summarize what it does? Is it from OpenSOC?
>> >
>> >
>> >
>> >
>> > On August 13, 2019 at 17:53:52, Nick Allen (n...@nickallen.org) wrote:
>> >
>> > As part of https://github.com/apache/metron/pull/1470, I found it
>> > difficult
>> > to update the "Least Recently Used Pruner" to work with HBase 2.0.2. I am
>> > sure that given more time and effort, I could make it work, but is it
>> worth
>> > it?
>> >
>> > This is a feature that I myself am not familiar with. I do not know of
>> > anyone using this. I also did not find much documentation on how to use
>> > this feature. I certainly don't know the entire user community, so please
>> > let me know if anyone is using this functionality or believes that it
>> > should be maintained going forward.
>> >
>> > Would you support deprecating this feature?
>> >
>> > Thanks
>> >
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
Re: [DISCUSS] HDP 3.1 Upgrade and release strategy
Given what Simon pointed out, where in your view are the current feature gaps with respect to that? We can back up the zookeeper config, any files we need on HDFS, and any custom parsers/enrichers that were produced. The rest of it is pretty much HDP-pecific upgrade. In my view, backing up kafka topic configs, HDFS configs, ES configs, etc., is outside of the scope of Metron and is more of a platform function. Thanks, James 27.08.2019, 15:56, "zeo...@gmail.com" : > I agree that having a scripted approach for backup and restore of Metron > configs should be necessary for such a large change/upgrade process. > Having been through this many times in the past I can tell you that the > difficulty of upgrading (whether perceived or actual) holds back adoption > of the platform. > > Jon Zeolla > > On Tue, Aug 27, 2019, 5:25 PM Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> Not sure it’s in the scope of the project to handle the HDP upgrade as >> well, I would scope it to metron config only, and point at the extensive >> upgrade capability of Ambari, rather than us trying to recreate the way the >> distribution works. >> >> Simon >> >> On Tue, 27 Aug 2019 at 22:23, Otto Fowler wrote: >> >> > If anyone can think of the things that need to be backed up, please >> > comment the jira. >> > >> > >> > >> > >> > On August 27, 2019 at 17:07:20, Otto Fowler (ottobackwa...@gmail.com) >> > wrote: >> > >> > Good idea METRON–2239 [blocker]. >> > >> > >> > >> > On August 27, 2019 at 16:30:13, Simon Elliston Ball ( >> > si...@simonellistonball.com) wrote: >> > >> > You could always submit a Jira :) >> > >> > On Tue, 27 Aug 2019 at 21:27, Otto Fowler >> wrote: >> > >> >> You are right, that is much better than backup_metron_configs.sh. >> >> >> >> >> >> >> >> On August 27, 2019 at 16:05:38, Simon Elliston Ball ( >> >> si...@simonellistonball.com) wrote: >> >> >> >> You can do this with zk_load_configs and Ambari’s blueprint api, so we >> >> kinda already do. >> >> >> >> Simon >> >> >> >> On Tue, 27 Aug 2019 at 20:19, Otto Fowler >> >> wrote: >> >> >> >>> Maybe we need some automated method to backup configurations and >> restore >> >>> them. >> >>> >> >>> >> >>> >> >>> On August 27, 2019 at 14:46:58, Michael Miklavcic ( >> >>> michael.miklav...@gmail.com) wrote: >> >>> >> >>> > Once you back up your metron configs, the same configs that worked on >> >>> the previous version will continue to work on the version running on >> HDP >> >>> 3.x. If there is any discrepancy between the two or additional >> settings >> >>> will be required, those will be documented in the release notes. From >> the >> >>> Metron perspective, this upgrade would be no different than simply >> >>> upgrading to the new Metron version. >> >>> >> >>> This upgrade cannot be performed the same way we've done it in the >> past. >> >>> A number of platform upgrades, including OS, are required: >> >>> >> >>> 1. Requires the OS to be updated on all nodes because there are no >> >>> Centos6 RPMs provided in HDP 3.1. Must bump to Centos7. >> >>> 2. The final new HBase code will not run on HDP 2.6 >> >>> 3. The MPack changes for new Ambari are not backwards compatible >> >>> 4. YARN and HDFS/MR are also at risk to be backwards incompatible >> >>> >> >>> >> >>> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic < >> >>> michael.miklav...@gmail.com> wrote: >> >>> >> >>>> Adding the dev list back into the thread (a reply-all was missed). >> >>>> >> >>>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota >> >>>> wrote: >> >>>> >> >>>>> I agree with Simon. HDP 2.x platform is rapidly approaching EOL and >> >>>>> everyone will likely need to migrate by end of year. Doing this >> platform >> >>>>> upgrade sooner will give everyone visibility into what Metron on HDP >> 3.x
Re: [DISCUSS] Upgrade to HDP 3.1.0
TableProvider abstraction and decided to try out a compromise >> approach that allows us to: >> >> 1. Upgrade as much of the API as possible in the current version of >> HBase against master >> 2. Manage connections within the TableProvider abstraction - this >> would have an API feel that is similar to what had been encapsulated by >> the >> HTableInterfaces we rely on currently, and remove a large chunk of the >> code >> that had been necessary to finish the upgrade. >> >> *Reducing Scope* >> >> We have long-lived connections to HBase that don't need to be >> opened/closed and pooled in the traditional request/response lifecycle >> sense. We know at the time our application spins up how many processes and >> threads there will be - this is static for us. I put together a PR ( >> https://issues.apache.org/jira/browse/METRON-2217) that migrates HTable >> and HTableInterface classes to the new Table API and encapsulates >> connection management within the TableProviders. We had some concerns about >> risks associated with managing the connections this way, as opposed to >> using a more robust connection management approach, so I reached out to the >> HBase community to get some guidance. The feedback we received suggests >> that managing our connections this way should be sufficient. And the HBase >> connection objects are threadsafe, to boot. >> >> https://lists.apache.org/thread.html/6b83cd7548efb8c37899063affc97e4c5ce823a13359a49b477e3c07@%3Cdev.hbase.apache.org%3E >> >> *A Revised Plan* >> >> The alternative HBase client/connection approach is promising, and it >> greatly reduces the overall architectural impact we will need to absorb >> alongside a major upgrade. The following is my proposal after some coding >> experiments and numerous conversations with Nick Allen, Ryan Merriman, >> Casey Stella, Otto Fowler, and James Sirota. >> >> 1. Do as much refactoring in small chunks as possible in master. e.g. >> the first phase of the HBase API changes. Reducing the overall number of >> variables changing all at the same time in the same place should reduce >> the >> overall risk of the upgrade. Prove stability with what we can in master >> and >> the issues we run into in the FB should be easier to isolate and solve. >> 2. Target the upgrade feature branch as being a place where we >> primarily have to deal with changes due to classpath problems. There will >> be some other necessary code changes, e.g. hbase coprocessor, however the >> changes should be well-isolated and narrower in scope. >> 3. Ryan and I have had numerous conversations surrounding the Maven >> dependency classpath issues that frequently come up at runtime anytime >> even >> the smallest change to our dependency tree occurs. I won't go into those >> details now, but you can see the discussion and history here ( >> https://github.com/apache/metron/pull/1436). While there's inherent >> risk in making big changes to our dep management, there is also a >> substantial upside - this PR makes finding classpath problems and solving >> them substantially easier. This PR is ready to go in master and should >> greatly speed up our ability to rectify and cp problems we encounter in >> the >> feature branch. >> 4. Find an analog for our port of the MockHTable ( >> >> https://github.com/apache/metron/blob/master/metron-platform/metron-hbase/metron-hbase-common/src/test/java/org/apache/metron/hbase/mock/MockHTable.java) >> in >> HBase 2.0.2. Nick has been working on a POC around this alongside my work >> on the other API migration and he has been able to get to a point with >> the >> integration tests passing. We had originally hoped this could be landed >> in >> master, but the underlying low-level supporting classes have changed and >> are not be forwards compatible the way the Table interface >> and ConnectionFactory class are. We plan to land this in the feature >> branch. >> 5. Manage component version changes in an HDP 3.1 profile that gets >> updated as PRs are submitted. This allows the modules to be upgraded on a >> per-component basis, while still compiling and allowing tests to run, >> without requiring a big bang all-or-nothing upgrade. We would then do a >> final reconciliation and deprecation of the old Hadoop versions and >> profile >> at the tail of the FB. >> https://issues.apache.org/ji
Re: [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images
i am ok with it as long as we are not forcing people to buy stuff 02.05.2019, 18:18, "Michael Miklavcic" : > Here's the latest discussion on the subject: > https://lists.apache.org/thread.html/0aa2b0b9ed4a0f0b0d8bb018c618e62de196565f9af71f347e504076@%3Cdev.metron.apache.org%3E > > I'd like to propose a vote to change our dev guidelines which will clarify > the tooling we use to produce diagrams and share the source files for those > diagrams. I propose the dev guidelines > https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines and > PR checklist > https://github.com/apache/metron/blob/master/.github/PULL_REQUEST_TEMPLATE.md#for-documentation-related-changes > be > changed in the following ways: > > 1. Under "1.1 Contributing A Code Change" > 1. Change <<"New features and significant bug fixes should be > documented in the JIRA and appropriate architecture diagrams should be > attached. Major features may require a vote.">> to <<"New features > and significant bug fixes should be documented in the JIRA. Appropriate > architecture diagrams should be created in https://www.draw.io/ > and committed > to source control as per section 2.4. Diagrams may be requested of PR > submitters during review either as documentation or as an aid to the > reviewer. Major features may also require a vote.">> > 2. Under "2.4 Documentation" > 1. New line item <<"Diagrams - We save architecture diagram source > files in an xml format rendered by draw.io (instructions below). This > is the free tool of choice that we've agreed to use for exchanging > diagrams and their source files in Metron.">> > 2. New line item < "/images-source" and rendered diagrams and images belong in > "/images." > 3. New subsection <<"Creating and Modifying Diagrams">>. This section > would provide basic instructions for downloading source files from > draw.io. > 3. Add a new checkbox item under PR checklist heading "For documentation > related changes" with the following text > 1. Have you ensured that any documentation diagrams have been > updated, along with their source files, using draw.io? See > > https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines > for > instructions. > 4. Here is the Jira for migrating/redoing existing diagrams > 1. https://issues.apache.org/jira/browse/METRON-2099 > > We require a minimum of 72 hours for a vote, not typically including > weekend days. I'd like to leave this vote open until Wednesday 5/8, 12PM > EDT. Please vote +1, -1, or 0 to abstain, and also indicate if your vote is > binding or non-binding. --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: [DISCUSS] Add ngrx to handle state management in Angular
I found a helpful article here: https://brianflove.com/2018/01/08/ngrx-the-basics/ A lot of this goes over my head, but in a nutshell, it's a tree-based state management object for JS. Its main drawback seems to be added complexity, but if the guys who are more familiar with UI say we would benefit from it I am inclined to take them at their word. 26.11.2018, 13:09, "Michael Miklavcic" : > Shane, thanks for sharing this. Can you perhaps describe a sample use case > in the UI currently and explain for us how it currently works (or doesn't, > ha) versus how it would be modified and improved with using NgRx? > > Thanks, > Mike > > On Fri, Nov 23, 2018 at 7:44 AM Shane Ardell > wrote: > >> What I'm referring to is roughly the entire contents of the UI >> application's memory. >> >> On Thu, Nov 22, 2018 at 6:29 PM Otto Fowler >> wrote: >> >> > Can you describe what you mean by “state” in a little more detail? Not a >> > complete description, maybe just a crib list. >> > >> > >> > On November 22, 2018 at 07:21:43, Shane Ardell (shane.m.ard...@gmail.com >> ) >> > wrote: >> > >> > As both the Management and Alerts UI grow in size, managing application >> > state continues to become more and more complex. To help us deal with >> > managing all of this state and ensuring our application derives state >> from >> > a single source of truth, I suggest we start using NgRx, a state >> > management >> > library based on the Redux pattern but built for Angular. It is by far >> the >> > most popular library of this type for Angular. As you can see in the >> > project's GitHub Insights tab <https://github.com/ngrx/platform/pulse>, >> > it's quite actively worked on and releases are pretty frequent. The >> > project >> > is licensed under MIT. >> > >> > As far as an approach to integration, I don't think we necessarily need a >> > big refactoring right off the bat. I feel something like this can be done >> > in a piecemeal approach over time. I think we can start by introducing it >> > into the project the next time we have a new application feature. >> > >> > What are everyone's thoughts around this? >> > >> > Cheers, >> > Shane >> > >> > --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
[ANNOUNCE] Shane Ardell is a committer
The Project Management Committee (PMC) for Apache Metron has invited Shane Ardell to become a committer and we are pleased to announce that he has accepted. I wanted to congratulate Shane on this achievement. Being a committer enables easier contribution to the project since there is no need to go via the patch submission process. This should enable better productivity. Being a PMC member enables assistance with the management and to guide the direction of the project. --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: [DISCUSS] Deprecating MySQL
To clarify my position, I don't have a problem with mySql or any other projects relying on it. mySql in itself is not an issue. What I don't want is for a customer to be presented with an option to chose and configure two options for authenticating the UI, which I think is needless. It adds complexity for not much value. Since LDAP is clearly the better way to go that should be what we support without explicitly giving a user an option to switch to JDBC. A user can still do so by extending our abstractions if that is what they chose to do, but this would not be officially supported by us. We would not be providing a config or an mPack to do this. A user would have to do it on their own. James 15.11.2018, 12:15, "Michael Miklavcic" : > Incidentally, even without the Metron piece in the picture, what is the > answer for Ambari's database dependency? Which uses a SQL data store. Does > this actually solve the problem of "customers won't install Metron bc SQL > store?" or are there other issues we need to address? > > On Thu, Nov 15, 2018 at 9:30 AM James Sirota wrote: > >> Hi Guys, >> >> My opinion on this, as is with Knox SSO, is that the code should be >> pluggable to support JDBC, but we should not continue to support the >> concrete implementation and expose it to users via a setting. This is a >> fairly minor feature and the added complexity of supporting switching >> between JDBC and LDAP is simply not worth it. We need to strike a balance >> between ease of use and capabilities/extensibility. For features that are >> worth it such as with analytics and stream processing, the extra capability >> is worth the added complexity in configuration. But for this, it is not. >> So let's keep JDBC around for a release to allow users to migrate to LDAP, >> deprecate it, and move on. >> >> Thanks, >> James >> >> 13.11.2018, 16:03, "Simon Elliston Ball" : >> > We went over the hbase user settings thing on extensive discussions at >> the time. Storing an arbitrary blob of JSON which is only ever accessed by >> a single key (username) was concluded to be a key value problem, not a >> relational problem. Hbase was concluded to be massive overkill as a key >> value store in this usecase, unless it was already there and ready to go, >> which in the case of Metron, it is, for enrichments, threat intel and >> profiles. Hence it ended up in Hbase, as a conveniently present data store >> that matched the usage patterns. See >> >> https://lists.apache.org/thread.html/145b3b8ffd8c3aa5bbfc3b93f550fc67e71737819b19bc525a2f2ce2@%3Cdev.metron.apache.org%3E >> and METRON-1337 for discussion. >> > >> > Simon >> > >> >> On 13 Nov 2018, at 18:50, Michael Miklavcic < >> michael.miklav...@gmail.com> wrote: >> >> >> >> Thanks for the write up Simon. I don't think I see any major problems >> with >> >> deprecating the general sql store. However, just to clarify, Metron >> does >> >> NOT require any specific backing store. It's 100% JPA, which means >> anything >> >> that can be configured with the Spring properties we expose. I think >> the >> >> most opinionated thing we do there is ship an extremely basic table >> >> creation script for h2 and mysql as a simple example for schema. As an >> >> example, we simply use H2 in full dev, which is entirely in-memory and >> spun >> >> up automatically from configuration. The recent work by Justin Leet >> removes >> >> the need to use a SQL store at all if you choose LDAP - >> >> https://github.com/apache/metron/pull/1246. I'll let him comment >> further on >> >> this, but I think there is one small change that could be made via a >> toggle >> >> in Ambari that would even eliminate the user from seeing JDBC settings >> >> altogether during install if they choose LDAP. Again, I think I'm on >> board >> >> with deprecating the SQL backing store as I pointed this out on the >> Knox >> >> thread as well, but I just wanted to make sure everyone has an accurate >> >> picture of the current state. >> >> >> >> I had to double check on the HBase config you mentioned, but it does >> appear >> >> that we use it for the Alerts UI. I don't think I realized we were >> storing >> >> config there instead of the Zookeeper store we use for other system >> >> configuration. Ironically enough, I think that it probably makes more &g
Re: [DISCUSS] Deprecate split-join enrichment topology in favor of unified enrichment topology
This is excellent work, Mike and long overdue. Thanks for doing this 05.11.2018, 16:46, "Michael Miklavcic" : > The PR has now been merged into master and closed. > > https://issues.apache.org/jira/browse/METRON-1855 > > On Sat, Nov 3, 2018 at 6:47 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> PR is out here - https://github.com/apache/metron/pull/1252 >> >> I made the unified enrichment topology the new default and marked the >> split-join topology as deprecated in various parts of the documentation. I >> think we should have a release with the deprecation notes and new default >> and then move to remove split-join entirely shortly thereafter. >> >> Best, >> Mike >> >> On Fri, Nov 2, 2018 at 5:47 AM Mohan Venkateshaiah < >> mvenkatesha...@hortonworks.com> wrote: >> >>> +1 (non-binding) >>> >>> Thanks >>> Mohan DV >>> >>> On 11/2/18, 3:29 PM, "zeo...@gmail.com" wrote: >>> >>> +1 totally agree. >>> >>> Jon >>> >>> On Fri, Nov 2, 2018, 1:31 AM Anand Subramanian < >>> asubraman...@hortonworks.com> >>> wrote: >>> >>> > Piling on my +1 (non-binding) as well. >>> > >>> > On 11/2/18, 4:41 AM, "Ryan Merriman" wrote: >>> > >>> > +1 >>> > >>> > On Thu, Nov 1, 2018 at 5:38 PM Casey Stella >> > >>> > wrote: >>> > >>> > > +1 >>> > > On Thu, Nov 1, 2018 at 18:34 Nick Allen >>> wrote: >>> > > >>> > > > +1 >>> > > > >>> > > > On Thu, Nov 1, 2018, 6:27 PM Justin Leet < >>> justinjl...@gmail.com> >>> > wrote: >>> > > > >>> > > > > +1, I haven't seen any case where the split-join topology >>> isn't >>> > made >>> > > > > obsolete by the unified topology. >>> > > > > >>> > > > > On Thu, Nov 1, 2018 at 6:17 PM Michael Miklavcic < >>> > > > > michael.miklav...@gmail.com> wrote: >>> > > > > >>> > > > > > Fellow Metronians, >>> > > > > > >>> > > > > > We've had the unified enrichment topology around for a >>> number >>> > of >>> > > months >>> > > > > > now, it has proved itself stable, and there is yet to >>> be a >>> > time that >>> > > I >>> > > > > have >>> > > > > > seen the split-join topology outperform the unified >>> one. Here >>> > are >>> > > some >>> > > > > > simple reasons to deprecate the split-join topology. >>> > > > > > >>> > > > > > 1. Unified topology performs better. >>> > > > > > 2. The configuration, especially for performance >>> tuning is >>> > much, >>> > > > much >>> > > > > > simpler in the unified model. >>> > > > > > 3. The footprint within the cluster is smaller. >>> > > > > > 4. One of the first activities for any install is >>> that we >>> > spend >>> > > time >>> > > > > > instructing users to switch to the unified topology. >>> > > > > > 5. One less moving part to maintain. >>> > > > > > >>> > > > > > I'd like to recommend that we deprecate the split-join >>> > topology and >>> > > > make >>> > > > > > the unified enrichment topology the new default. >>> > > > > > >>> > > > > > Best, >>> > > > > > Mike >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>> > >>> > -- >>> >>> Jon Zeolla --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: [DISCUSS] Deprecating MySQL
just storing a table of users >>> (Eclipse Link, JPA, the MySQL server and the need to get clients installed >>> and pushed separately because of licence requirements) >>> * Organisations don't want to have to manage yet another user source of >>> truth since this leads to operational complexity. >>> >>> In short, managing our own user store makes very little sense to operations >>> users. >>> >>> Some of these (licence and inconsistency for example) could be solved by >>> changing our default DB to something like Postgres, which has easier terms >>> to deal with. We could start encrypting passwords, but there would still be >>> a lot of dependencies to store users, which is a problem much better solved >>> by LDAP. >>> >>> Now that we have the option to use LDAP for user storage, I would suggest >>> that we deprecate and ultimately remove all the RDBMS and ORM dependencies, >>> which significantly reduces our dependencies and simplifies deployment and >>> long term management of Metron clusters. >>> >>> So I propose that we deprecate the RDBMS use in the next Apache release, >>> and then strip out the RDBMS stuff in the following. We would continue to >>> use LDAP for users and HBase for non-LDAPy user settings (as we currently >>> do). We should also provide a small demo LDAP for full dev. Since we are >>> looking at adding Knox into the stack, that project provides a convenient >>> mini-LDAP demo service which would do this job without the need to add >>> additional components. >>> >>> Thoughts? Anyone relying on MySQL for users (if so, are you aware that your >>> passwords are all plaintext? How do you currently handle the shortcomings >>> and admin overhead?) Any objections? >>> >>> Simon --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: [DISCUSS] Knox SSO feature branch review and features
gt;> > >> > > operative question is can we do the feature without making the >> OTHER >> > >> > > architectural change >> > >> > > (e.g. migrating from expressjs to spring boot + zuul). I would >> > argue >> > >> > that >> > >> > > if we WANT to do that, then it should be a separate feature >> branch. >> > >> > > >> > >> > > Thus, I leave with a question: is there a way to accomplish this >> > >> feature >> > >> > > without ripping out expressjs? >> > >> > > >> > >> > > - If so and it is feasible, I would argue that we should >> decouple >> > >> this >> > >> > > into a separate feature branch. >> > >> > > - If so and it is infeasible, I'd like to hear an argument as >> to >> > >> the >> > >> > > infeasibility and let's decide given that >> > >> > > - If it is not possible, then I'd argue that we should keep >> them >> > >> > coupled >> > >> > > and move this through as-is. >> > >> > > >> > >> > > On a side-note, it feels a bit weird that we're narrowing to a >> > bundled >> > >> > > proxy, rather than having that be a pluggable thing. I'm not >> super >> > >> > > knowledgeable in this space, so I apologize >> > >> > > in advance if this is naive, but isn't this a pluggable, external >> > >> > component >> > >> > > (e.g. nginx)? >> > >> > > >> > >> > > On Thu, Sep 27, 2018 at 5:05 PM Michael Miklavcic < >> > >> > > michael.miklav...@gmail.com> wrote: >> > >> > > >> > >> > > > I've spent some more time reading through Simon's response and >> the >> > >> > added >> > >> > > > sequence diagram. This is definitely helpful - thank you Simon. >> > >> > > > >> > >> > > > I need to redact my initial list: >> > >> > > > >> > >> > > > 1. Node migrated to Spring Boot, expressjs migrated to a >> > >> > > > non-JS/non-NodeJs proxying mechanism (ie Zuul in this case) >> > >> > > > 2. JDBC removed completely in favor of LDAP >> > >> > > > 3. Knox/SSO >> > >> > > > >> > >> > > > I'm a bit conflicted on the best way to move forward and would >> > like >> > >> > some >> > >> > > > thoughts from other community members on this. I think an >> argument >> > >> can >> > >> > be >> > >> > > > made that 1 and 2 are independent of 3, and should/could really >> be >> > >> > > > independent PR's against master. >> > >> > > > >> > >> > > > The need for a replacement for expressjs (Zuul in this case) is >> an >> > >> > > artifact >> > >> > > > that our request/response cycle for REST calls is a simple >> matter >> > of >> > >> > > > forwarding with some additional headers for authentication. >> > There's >> > >> a >> > >> > > > JSESSIONID managed by the client browser in our current >> > >> architecture, >> > >> > for >> > >> > > > example. You login to the alerts or the management UI which >> > >> forwards a >> > >> > > > request to REST, which looks up credentials in a backend >> database, >> > >> and >> > >> > > > passes the results back up the chain. All browser requests go >> > >> directly >> > >> > to >> > >> > > > the specific UI you're working with - this is the CORS problem. >> > You >> > >> > > can't, >> > >> > > > without some effort with headers for adding other domains to the >> > >> safe >> > >> > > list >> > >> > > > or disabling the security check for CORS, make remote calls >> > >> directly to >> > >> > > > REST. Th
Re: [DISCUSS] Batch Profiler Feature Branch
; 2. The batch profiler process would be to take that >> > >> exact >> > >> > >> > profile >> > >> > >> > > > > definition from ZK and run the batch loader with >> that >> > >> from >> > >> > >> the >> > >> > >> > > CLI. >> > >> > >> > > > > 3. Profiles are now up to date from t0 - tCurrent >> > >> > >> > > > > 2. I've already done #1 above. Time goes by and now I >> > >> want to >> > >> > >> add >> > >> > >> > a >> > >> > >> > > > new >> > >> > >> > > > > profile. >> > >> > >> > > > > 1. Same first step above >> > >> > >> > > > > 2. I would run the batch loader with *only* that >> new >> > >> > profile >> > >> > >> > > > > definition to seed? >> > >> > >> > > > > >> > >> > >> > > > > Forgive me if I missed this in PR's and discussion in the >> > FB, >> > >> > but >> > >> > >> how >> > >> > >> > > do >> > >> > >> > > > we >> > >> > >> > > > > establish "tm" from 1.1 above? Any concerns about overlap >> > or >> > >> > gaps >> > >> > >> > after >> > >> > >> > > > the >> > >> > >> > > > > seeding is performed? >> > >> > >> > > > > >> > >> > >> > > > > On Thu, Sep 20, 2018 at 10:26 AM Nick Allen < >> > >> n...@nickallen.org >> > >> > > >> > >> > >> > > wrote: >> > >> > >> > > > > >> > >> > >> > > > > > I think more often than not, you would want to load >> your >> > >> > profile >> > >> > >> > > > > definition >> > >> > >> > > > > > from a file. This is why I considered the 'load from >> Zk' >> > >> more >> > >> > >> of a >> > >> > >> > > > > > nice-to-have. >> > >> > >> > > > > > >> > >> > >> > > > > > - In use case 1 and 2, this would definitely be the >> > >> case. >> > >> > >> The >> > >> > >> > > > > profiles >> > >> > >> > > > > > I am working with are speculative and I am using the >> > >> batch >> > >> > >> > > profiler >> > >> > >> > > > to >> > >> > >> > > > > > determine if they are worth keeping. In this case, >> my >> > >> > >> > speculative >> > >> > >> > > > > > profiles >> > >> > >> > > > > > would not be in Zk (yet). >> > >> > >> > > > > > - In use case 3, I could see it go either way. It >> > >> might be >> > >> > >> > useful >> > >> > >> > > > to >> > >> > >> > > > > > load from Zk, but it certainly isn't a blocker. >> > >> > >> > > > > > >> > >> > >> > > > > > >> > >> > >> > > > > > > So if the config does not correctly match the >> profiler >> > >> > config >> > >> > >> > held >> > >> > >> > > in >> > >> > >> > > > > ZK >> > >> > >> > > > > > and >> > >> > >> > > > > > the user runs the batch seeding job, what happens? >> > >> > >> > > > > > >> > >> > >> > > > > > You would just get a profile that is slightly different >> > >> over >> > >> > the >> > >> > &g
Re: [DISCUSS] Knox SSO feature branch review and features
; > > https://issues.apache.org/jira/browse/METRON-1755, the architecture >>> > > diagram. Minimally, an explanation of the current architecture along >>> with >>> > > discussion around the additional proposed changes and rationale would >>> be >>> > > useful for evaluation. I don't have a solid enough understanding yet >>> of >>> > the >>> > > full scope of changes and how they differ from the existing >>> architecture >>> > > just from looking at the PR's alone. >>> > > >>> > > 1. The first question is a general one regarding the necessity of >>> the >>> > 3 >>> > > additional features alongside Knox - migrating Node to Spring Boot, >>> > > removing JDBC altogether, adding dependencies on Netflix's Zuul >>> > > framework. >>> > > Are these necessary for adding Knox/SSO? They seem like potentially >>> > > separate features, imo. >>> > > 2. It looks like LDAP will be a required component for interacting >>> > with >>> > > Metron via the UI's. I see this PR >>> > > https://github.com/apache/metron/pull/1186 which removes JDBC >>> > > authentication. Are we ready to remove it completely or would it be >>> > > better >>> > > to leave it as a minimal installation option? What is the proposed >>> > > migration path for existing users? Do we feel comfortable requiring >>> > that >>> > > all installations, including full dev, install and configure LDAP? >>> For >>> > > comparison, in the PCAP feature branch we discussed removing the >>> > > existing >>> > > PCAP REST application in the initial discussion, got agreement, and >>> > > later >>> > > removed it in the course of working on the feature branch. The PR >>> is >>> > > fairly >>> > > clear, however I think we're just missing some basic discussion >>> around >>> > > the >>> > > implications, as I've outlined above. Some additional relevant >>> > > discussion >>> > > occurred on this PR https://github.com/apache/metron/pull/1112 >>> which >>> > > would be good to summarize for purposes of this overarching >>> > architecture >>> > > discussion. >>> > > 3. Migration from Node to Spring Boot. I believe this is already >>> used >>> > by >>> > > the REST application and if anything brings some cohesion to our >>> > server >>> > > strategy. Strictly speaking, is there a reason this is required for >>> > > Knox? >>> > > It seems comparable to a component upgrade, such as moving from ES >>> 2.x >>> > > to >>> > > 5.6.x and upgrading Angular 6. >>> > > 4. Introduction of Netflix's Zuul. >>> > > https://issues.apache.org/jira/browse/METRON-1665. >>> > > - > "The UIs currently proxy to the REST API to avoid CORS >>> issues, >>> > > this will be achieved with Zuul." >>> > > - Can we elaborate more on where or how CORS is a problem with >>> our >>> > > existing architecture, how Zuul will help solve that, and how it >>> > > fits with >>> > > Knox? Wouldn't this be handled by Knox? Since Larry McCay >>> chimed in >>> > > with >>> > > interest on the original SSO thread about the FB, I'm hoping he >>> is >>> > > also >>> > > willing to chime in on this as well. >>> > > - This looks like it has the potential to be a rather large >>> piece >>> > of >>> > > fundamental infrastructure (as it's also pertinent to >>> > microservices) >>> > > to >>> > > pull into the platform, and I'd like to be sure the community is >>> > > aware of >>> > > and is OK with the implications. >>> > > 5. > "The proposal is to use a spring boot application, allowing >>> us to >>> > > harmonize the security implementation across the UI static servers >>> and >>> > > the >>> > > REST layer, and to provide a routing platform for later >>> > microservices." >>> > > - >>> > > https://issues.apache.org/jira/browse/METRON-1665. >>> > > - Microservices is a pretty loaded term. I know there had been >>> some >>> > > discussion a while back during the PCAP feature branch start, >>> but I >>> > > don't >>> > > recall ever reaching a consensus on it. More detail in this >>> thread >>> > - >>> > > >>> > > >>> > >>> >>> https://lists.apache.org/thread.html/1db7c6fa1b0f364f8c03520db9989b4f7a446de82eb4d9786055048c@%3Cdev.metron.apache.org%3E >>> > > . >>> > > Can we get some clarification on what is meant by microservices >>> > > in the case >>> > > of this FB and relevant PR's, what that architecture looks like, >>> > and >>> > > how >>> > > it's achieved with the proposed changes in this PR/FB? It seems >>> > Zuul >>> > > is >>> > > also pertinent to this discussion, but there are many ways to >>> > > skin this cat >>> > > so I don't want to presume - >>> > > >>> > > >>> https://blog.heroku.com/using_netflix_zuul_to_proxy_your_microservices >>> > > 6. Zuul, Spring Boot, and microservices - Closely related to >>> > point 5 >>> > > above. It seems that we weren't quite ready for this when it was >>> > > brought up >>> > > in May, or at the very least we had some concern of what direction >>> to >>> > > go. >>> > > What is the operational impact, mpack impact, and how we propose to >>> > > manage >>> > > it with Kerberos, etc.? >>> > > >>> > > >>> > >>> >>> https://lists.apache.org/thread.html/c19904681e6a6d9ea3131be3d1a65b24447dca31b4aff588b263fd87@%3Cdev.metron.apache.org%3E >>> > > >>> > > There is a lot to like in this feature branch, imo. Great feature >>> > addition >>> > > with Knox and SSO. Introduction of LDAP support for authentication for >>> > > Metron UI's. Simplification/unification of our server hosting >>> > > infrastructure. I'm hoping we can flesh out some of the details >>> pointed >>> > out >>> > > above a bit more and get this feature through. Great work so far! >>> > > >>> > > Best, >>> > > Mike Miklavcic >>> > > >>> > >> >> -- >> -- >> simon elliston ball >> @sireb > > -- > -- > simon elliston ball > @sireb --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: [DISCUSS] Batch Profiler Feature Branch
t path to `hdfs://localhost:8020/...` to > match the port defined by HDP, instead of port 9000. > > [1] https://issues.apache.org/jira/browse/METRON-1699 > [2] > https://lists.apache.org/thread.html/da81c1227ffda3a47eb2e5bb4d0b162dd6d36006241c4ba4b659587b@%3Cdev.metron.apache.org%3E > [3] > https://lists.apache.org/thread.html/d28d18cc9358f5d9c276c7c304ff4ee601041fb47bfc97acb6825083@%3Cdev.metron.apache.org%3E --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: [DISCUSS] Migrate from Protractor to Cypress
This article comparing the two is not favorable for Cypress. Are any of these concerns relevant to us? If not, then I think Cypress is fine https://hackernoon.com/cypress-io-vs-protractor-e2e-testing-battle-d124ece91dc7
Re: [DISCUSS] PCAP data for testing and development
I think another reason why we removed it was that it was being flagged by antivirus tools. I am not sure that loop and stop would do anything because the resources would still be taken up by idle topologies and idle sensors. I think when we switch to containers and don't have to eat the overhead of the VM we can potentially do something like this. I think the PCAP feature development is more or less coming to completion so I am not sure it's worth investing into this.
Re: [DISCUSS] Internal Metron fields
I propose that we just disallow having dots in the field name. Dots seem to have a special meaning and as we keep adding data stores we may run into some unintended behavior. We should have logic in our code to check for it and either auto-correct it (replace with underscores?) or at least throw an error or a warning. Thanks, James 07.09.2018, 16:33, "Ryan Merriman" : > Internal means it’s not configurable, doesn’t contain our default separator > (dots) and is namespaced with metron. We can definitely improve on DRY but > there’s more to it than that. For example, having 2 different versions of > this field name (ES and Solr) adds a significant amount of complexity for no > real benefit. > >> On Sep 7, 2018, at 5:12 PM, Michael Miklavcic >> wrote: >> >> Can you elaborate on what you mean by "convert to internal?" From your >> description, it looks like the challenge is from our violations of DRY when >> it comes to constants referencing those keys, which would be eliminated by >> refactoring. >> >>> On Fri, Sep 7, 2018, 3:50 PM Ryan Merriman wrote: >>> >>> I recently worked on a PR that involved changing the default behavior of >>> the ElasticsearchWriter to store data using field names with the default >>> Metron separator, dots. One of the unfortunate consequences of this is >>> that although dots are allowed in more recent versions of ES, it changes >>> how these fields are stored. Having a dot in a field name causes ES to >>> treat it as an object field type. We're not quite comfortable with this >>> because it could introduce unforeseen side effects that may not be >>> obvious. Here's the PR: https://github.com/apache/metron/pull/1181 >>> >>> As I worked through it I noticed there are a couple fields that include >>> separators where it's not actually necessary. They are not nested by >>> nature and are internal to Metron. The fact that they are internal means >>> they show up in constants and are hardcoded in several different places. >>> That made the work in the PR above much harder and tedious than it should >>> have been. There are 2 in particular that I had to deal with: source:type >>> and threat:triage:score in metaalerts. >>> >>> Is it worth considering converting these to internal Metron fields so that >>> they stay constant and this isn't a problem in the future? I could see >>> these fields following the same pattern as 'metron_alert'. However this >>> would cause pain when upgrading because existing data would need to be >>> updated with these new fields. >>> >>> Just an idea. Curious if other have an opinion on the subject. --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: [VOTE] Metron Release Candidate 0.6.0-RC1
+1 (binding) ran it up in full dev 07.09.2018, 06:30, "Justin Leet" : > This is a call to vote on releasing Apache Metron 0.6.0 and the Bro-Kafka > plugin 0.2.0 > > Full list of changes in this release: > https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/CHANGES > https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/CHANGES.bro-plugin > > The tags to be voted upon are: > (apache/metron) apache-metron-0.6.0-rc1 > (apache/metron-bro-plugin-kafka) apache-metron-bro-plugin-kafka_0.2.0-rc1 > > The source archives being voted upon can be found here: > https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/apache-metron-0.6.0-rc1.tar.gz > https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/apache-metron-bro-plugin-kafka_0.2.0-rc1.tar.gz > > Other release files, signatures and digests can be found here: > https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/ > > The release artifacts are signed with the following key: > https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/KEYS > > Please vote on releasing this package as Apache Metron 0.6.0-RC1 and the > Bro-Kafka plugin as 0.2.0-RC1 > > When voting, please list the actions taken to verify the release. > > Recommended build validation and verification instructions are posted > here: > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds > > This vote will be open for until 10pm EDT on Wednesday September 12 2018, > to account for the weekend. > > [ ] +1 Release this package as Apache Metron 0.3.0-RC1 > > [ ] 0 No opinion > > [ ] -1 Do not release this package because... --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: [DISCUSS] Pcap query branch completion
+1 on the merge as well 16.08.2018, 05:46, "Casey Stella" : > I'm +1 on the merge. This is great work and congrats to those who > contributed to it! > > On Thu, Aug 16, 2018 at 8:27 AM Otto Fowler wrote: > >> Looks good, thanks! >> >> On August 15, 2018 at 19:38:12, Ryan Merriman (merrim...@gmail.com) wrote: >> >> Otto, I believe the items you requested are in the feature branch now. Is >> there anything outstanding that we missed? The Jiras for the Pcap feature >> branch should be up to date: >> https://issues.apache.org/jira/browse/METRON-1554 >> >> On Mon, Aug 13, 2018 at 5:13 PM, Ryan Merriman >> wrote: >> >> > - Date range limits on queries >> > >> > I will add a warning in the Job cleanup PR. That seems like an >> > appropriate place for it (ie. make sure you don't cause health issues in >> > your cluster). >> > >> > - UI should manage a queue/history of jobs >> > >> > I can add some documentation around killing jobs manually with the YARN >> > CLI. However if they haven't set up a YARN queue, I'm not sure how you >> > would view only Pcap jobs. I'm also not sure how you would get the >> > application id for the job to kill because it's not displayed anywhere in >> > the UI. However, I believe we are wired for a job name but REST doesn't >> > set this. Maybe we could get a proper job name associated with pcap >> > queries and then this would be possible to document? >> > >> > - Documentation/blueprint for YARN configuration >> > >> > You make a good point. A YARN tuning guide for Metron does sound useful. >> > I will add a follow on Jira. >> > >> > On Mon, Aug 13, 2018 at 4:53 PM, Otto Fowler >> > wrote: >> > >> >> >> >> - Date range limits on queries >> >> >> >> I took the point the wrong way apparently, sorry, I withdraw. I thought >> >> you meant allow specifying a limit on the query, not the system imposing >> a >> >> limit. >> >> This should be documented with a warning or something >> >> >> >> - UI should manage a queue/history of jobs >> >> >> >> I was thinking that if there where multiple users/jobs, there should >> >> be some thought or documentation + script on how to manage them. >> >> “To see all the jobs still running on your cluster, across users and ui >> >> instances do X” >> >> “If there is an issue with the jobs you can’t resolve in the UI for that >> >> user, or you are an admin and want to do something then X" >> >> >> >> - Documentation/blueprint for YARN configuration >> >> >> >> I agree with what you are saying. Although, we offer guidance on storm >> >> tuning, and that is conceptually the same isn’t it? That is why it comes >> >> to mind. >> >> Maybe this can be a follow on, in the tuning guide? >> >> >> >> On August 13, 2018 at 17:36:41, Ryan Merriman (merrim...@gmail.com) >> >> wrote: >> >> >> >> - Date range limits on queries >> >> >> >> Can you describe what you think is needed here? Each Metron user could >> >> have different volumes of pcap data spread out over different time >> >> periods. Are you saying we should limit the data range to something >> either >> >> >> >> constant or configurable? Are we sure all users would want this? Am I >> >> misinterpreting this requirement? >> >> >> >> - UI should manage a queue/history of jobs >> >> >> >> What should we document here? Reading that bullet point again, it's sort >> >> of vague and not very description. What I am referring to is a design >> that >> >> >> >> provides users a way to view and manage jobs in the UI. Currently jobs >> can >> >> >> >> only be run 1 at a time and progress is shown with a status bar, so it's >> >> somewhat interactive. >> >> >> >> - Documentation/blueprint for YARN configuration >> >> >> >> >> > --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: Pcap Query Panel feature branch status
I had a chance to look at the feature branch recently and I think we should merge it into master as soon as possible to avoid further diversion between the two branches. In my opinion the feature is complete enough to be merged into the main project. Thanks, James 07.08.2018, 17:23, "Michael Miklavcic" : > I'd like to put up a DISCUSS thread in the next day or so regarding getting > the pcap feature branch merged. In preparation, I am going to share an > accounting of completed and outstanding tasks. Can folks that have > contributed update their Jira status in the subtasks? It looks like the > current state of affairs is a bit outdated and I'd like to have this > buttoned up before we officially present this to the community. > > https://issues.apache.org/jira/browse/METRON-1554 > > Also, any Jiras that have been created that are relevant to the feature > branch but have not been made subtasks should be converted. > > - Open the Jira > - select "More" > - choose "convert to subtask." > - Search for METRON-1554 in the search box and select the Pcap epic that > shows up. > > Thanks, > Mike --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: [DISCUSS] Metron Parsers in Nifi
i. We should make that separate. Running Stellar over Records would be >>> the best thing. >>> >>> - This Processor would work similarly to Storm: bytes[] in -> JSON >>> out. >>> - There is a Processor >>> < >>> >>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JoltTransformJSON.java >>> > >>> that >>> handles loading other JARs that we can model a >>> MetronParserProcessor off of >>> that handles classpath/classloader issues (basically just sets up a >>> classloader specific to what's being loaded and swaps out the Thread's >>> loader when it calls to outside resources). >>> >>> There should be no reason to load modules outside the NAR. Why do you >>> expect to? If each Metron Processor equiv of a Metron Storm Parser is just >>> parsing to json it shouldn’t need much.And we could package them in the >>> NAR. I would suggest we have a Processor per Parser to allow for >>> specialization. It should all be in the nar. >>> >>> The Stellar Processor, if you would support the works would possibly need >>> this. >>> >>> 3. Create a MetronZkControllerService to supply our configs to our >>> processors. >>> - This is a pretty established NiFi pattern for being able to provide >>> access to other services needed by a Processor (e.g. databases or large >>> configurations files). >>> - The same controller service can be used by all Processors to manage >>> configs in a consistent manner. >>> >>> I think controller services would make sense where needed, I’m just not >>> sure what you imagine them being needed for? >>> >>> If the user has NiFi, and a Registry etc, are you saying you imagine them >>> using Metron + ZK to manage configurations? Or to be using BOTH storm >>> processors and Nifi Processors? >>> >>> At that point, we can just NAR our controller service and parser processor >>> >>> up as needed, deploy them to NiFi, and let the user provide a config for >>> where their custom parsers can be provided (i.e. their parser jar). This >>> would be 3 nars (processor, controller-service, and controller-service-api >>> >>> in order to bind the other two together). >>> >>> Once deployed, our ability to use parsers should fit well into the >>> standard >>> NiFi workflow: >>> >>> 1. Create a MetronZkControllerService. >>> 2. Configure the service to point at zookeeper. >>> 3. Create a MetronParser. >>> 4. Configure it to use the controller service + parser jar location + >>> any other needed configs. >>> 5. Use the outputs as needed downstream (either writing out to Kafka or >>> feeding into more MetronParsers, etc.) >>> >>> Chaining parsers should ideally become a matter of chaining MetronParsers >>> >>> (and making sure the enveloping configs carry through properly). For >>> parser >>> aggregation, I'd just avoid it entirely until we know it's needed in NiFi. >>> >>> Justin --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: [DISCUSS] Merging Solr feature branch (METRON-1416) into master
gt;>> > > > - METRON-1632 <https://issues.apache.org/jira/browse/METRON-1632> >>>> - >>>> > > > Causes a metaalert specific issue where UI filtering on >>>> > > > source.type:metaalert fails. More detail is on the Jira. >>>> > > > - Two validation tickets. It's been run up on multinode, and manual >>>> > > > testing has happened (and I'm will be seen a bit more on the final >>>> > PR >>>> > > by >>>> > > > various reviewers), so I'm inclined to just leave these open until >>>> > > we're >>>> > > > good to go. Let me know if we want to handle this differently. >>>> > > > >>>> > > > I'm of the opinion both of the active PRs need to be merged before >>>> we >>>> > > merge >>>> > > > this into master, especially the documentation one. The other two >>>> > > tickets >>>> > > > can be done in the future; one can be worked around and one is a >>>> > > metaalert >>>> > > > specific issue that primarily effects the alerts UI. >>>> > > > >>>> > > > As the branch has grown and diverged from master, it's gotten >>>> > > increasingly >>>> > > > unwieldy to maintain (and I think it's worth a follow-on discussion >>>> > about >>>> > > > how we manage refactorings that happen in these sorts of >>>> branches). I >>>> > > know >>>> > > > there's been at least a couple merges from master that have been >>>> > > > nontrivially difficult and required careful testing, particularly >>>> > around >>>> > > > the DAO layer, to avoid regressions in both code and tests. >>>> > > > >>>> > > > The feature set is pretty complete. The UI works, barring the >>>> > metaalert >>>> > > > issue. Much of the backend has been refactored and seen improved >>>> test >>>> > > > coverage benefiting both Solr and Elasticsearch. The main >>>> difference >>>> > > > between ES and Solr is the lack of the equivalent visualizations to >>>> > > > Kibana. I don't believe the feature branch needs to wait for this, >>>> as >>>> > > it's >>>> > > > pretty standalone work that can be added as usage and demand >>>> dictates. >>>> > > > >>>> > > > I'm of the opinion that the benefits of getting the branch into >>>> master >>>> > > > outweighs the issues still present, especially in terms of making >>>> > > > refactoring and features available and easing the dev burden. The >>>> > > > remaining tickets are Solr specific, and ES functions as it does in >>>> > > master. >>>> > > > >>>> > > > Are there any must-haves before we bring this branch back? Are >>>> there >>>> > any >>>> > > > other concerns we have before a final PR is opened (pending >>>> completion >>>> > of >>>> > > > active PRs and any other must-haves)? >>>> > > > >>>> > > > Justin >>>> > > > >>>> > > >>>> > --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
CVE-2018-1273 fixed in Metron 0.5.0
The following CVE was fixed in Metron 0.5.0: [CVEID]: CVE-2018-1273 [PRODUCT]:Spring Data Commons [VERSION]: versions prior to 1.13 to 1.13.10, 2.0 to 2.0.5, and older [PROBLEMTYPE]:remote code execution attack [REFERENCES]: https://pivotal.io/security/cve-2018-1273 [DESCRIPTION]: Spring Data Commons, versions prior to 1.13 to 1.13.10, 2.0 to 2.0.5, and older unsupported versions, contain a property binder vulnerability caused by improper neutralization of special elements. An unauthenticated remote malicious user (or attacker) can supply specially crafted request parameters against Spring Data REST backed HTTP resources or using Spring Data’s projection-based request payload binding hat can lead to a remote code execution attack.
Re: [DISCUSS] Update Metron Elasticsearch index names to metron_
i am +1 on it then
26.01.2018, 15:56, "Michael Miklavcic" <michael.miklav...@gmail.com>:
> Just checked on the length issue - we should be good -
> https://github.com/elastic/elasticsearch/issues/8079
>
> On Fri, Jan 26, 2018 at 3:37 PM, James Sirota <jsir...@apache.org> wrote:
>
>> Seems reasonable to me. The only thing is that it may make the index
>> names too long. Not sure if that matters to ES or not
>>
>> 26.01.2018, 15:32, "Simon Elliston Ball" <si...@simonellistonball.com>:
>> > +1 on this. The idea of a default broad matching template should also
>> include an order entry to avoid conflicts with more specific templates, and
>> we should then document the need for a higher order value in all per-source
>> index templates.
>> >
>> > In terms of production migration, I think we may want to provide some
>> detailed documentation in the upgrade guide on this, because there will be
>> people with a lot of existing indices that will be difficult to handle. We
>> may also need some tooling, but I expect docs would do the job. What do
>> people think about migration?
>> >
>> > Simon
>> >
>> >> One other benefit of this revised approach - we can more effectively
>> use
>> >> index template patterns to specify our base set of Metron property
>> types.
>> >> Call me crazy, but I think we should be able to do something like:
>> >>
>> >>
>> >>
>> >> {
>> >> *"template": "metron_*",*
>> >> "mappings": {
>> >> "metron_doc": {
>> >> "dynamic_templates": [
>> >> {
>> >> "geo_location_point": {
>> >> "match": "enrichments:geo:*:location_point",
>> >> "match_mapping_type": "*",
>> >> "mapping": {
>> >> "type": "geo_point"
>> >> }
>> >> }
>> >> },
>> >> {
>> >> "geo_country": {
>> >> "match": "enrichments:geo:*:country",
>> >> "match_mapping_type": "*",
>> >> "mapping": {
>> >> "type": "keyword"
>> >> }
>> >> }
>> >> },
>> >> {
>> >> "geo_city": {
>> >> "match": "enrichments:geo:*:city",
>> >> "match_mapping_type": "*",
>> >> "mapping": {
>> >> "type": "keyword"
>> >> }
>> >> }
>> >> },
>> >> {
>> >> "geo_location_id": {
>> >> "match": "enrichments:geo:*:locID",
>> >> "match_mapping_type": "*",
>> >> "mapping": {
>> >> "type": "keyword"
>> >> }
>> >> }
>> >> },
>> >> {
>> >> "geo_dma_code": {
>> >> "match": "enrichments:geo:*:dmaCode",
>> >> "match_mapping_type": "*",
>> >> "mapping": {
>> >> "type": "keyword"
>> >> }
>> >> }
>> >> },
>> >> {
>> >> "geo_postal_code": {
>> >> "match": "enrichments:geo:*:postalCode",
>> >> "match_mapping_type": "*",
>> >> "mapping": {
>> >> "type": "keyword"
>> >> }
>> >> }
>> >> },
>> >> {
>> >> "geo_latitude": {
>> >> "match": "enrichments:geo:*:latitude",
>> >> "match_mapping_type": "*",
>> >> "mapping": {
>> >> "type": "float"
>> >> }
>> >> }
>> >> },
>> >> {
>> >> "geo_longitude": {
>> >> "match": "enrichments:geo:*:longitude",
>> >> "match_mapping_type": "*",
>> >> "mapping": {
>> >> "type": "float"
>> >> }
>> >> }
>> >> },
>> >> {
>> >> "timestamps": {
>> >> "match": "*:ts"
Re: [DISCUSS] Time to remove github updates from dev?
Should we file an infra ticket on this? 19.01.2018, 13:56, "zeo...@gmail.com" <zeo...@gmail.com>: > I would give that +1 as well. > > Jon > > On Fri, Jan 19, 2018 at 3:32 PM Casey Stella <ceste...@gmail.com> wrote: > >> I could get behind that. >> >> On Fri, Jan 19, 2018 at 3:31 PM, Andre <andre-li...@fucs.org> wrote: >> >> > Folks, >> > >> > May I suggest Metron follows the NiFi mailing list strategy (we got >> > inspired by another project but I don't recall the name) and remove the >> > github comments from the dev list? >> > >> > Within NiFi we have both the dev and the issues lists. dev is for humans, >> > issues is for JIRA and github commits.[1] >> > >> > This allows the list thread list to be cleaner and is particularly >> helpful >> > for those reading the list from a list aggregation service. >> > >> > Cheers >> > >> > >> > [1] https://lists.apache.org/list.html?iss...@nifi.apache.org >> > > > -- > > Jon --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: [DISCUSS] Update Metron Elasticsearch index names to metron_
>> },
>> {
>> "threat_triage_reason": {
>> "mapping": {
>> "type": "text",
>> "fielddata": "true"
>> },
>> "match": "threat:triage:rules:*:reason",
>> "match_mapping_type": "*"
>> }
>> },
>> {
>> "threat_triage_name": {
>> "mapping": {
>> "type": "text",
>> "fielddata": "true"
>> },
>> "match": "threat:triage:rules:*:name",
>> "match_mapping_type": "*"
>> }
>> }
>>
>> ]}}
>>
>> That means that for every new sensor we bring on board we can skip
>> adding that boiler plate mapping config to every new template.
>>
>> On Wed, Jan 24, 2018 at 6:34 PM, Michael Miklavcic <
>> michael.miklav...@gmail.com> wrote:
>>
>>> I hear you Ali. I think this type of change would actually ease issues
>>> with downtime because it offers an easy path to migrating existing indices.
>>> I'd have to review the specifics in the ES docs again, but I believe you
>>> could duplicate the old indexes and migrate them to "metron_" in advance of
>>> the upgrade, and then consume new data to the new index pattern/name after
>>> the upgrade. That should be pretty seamless, I think. I guess it depends on
>>> how you're using ES.
>>>
>>> On Wed, Jan 24, 2018 at 4:08 PM, Ali Nazemian <alinazem...@gmail.com>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I just wanted to say it would be great if we can be careful with these
>>>> type
>>>> of changes. From the development point of view, it is just a few lines of
>>>> code which can provide multiple advantages, but for live large-scale
>>>> Metron
>>>> platforms, some of these changes might be really expensive to address with
>>>> zero-downtime.
>>>>
>>>> Cheers,
>>>> Ali
>>>>
>>>> On Thu, Jan 25, 2018 at 9:29 AM, Otto Fowler <ottobackwa...@gmail.com>
>>>> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> On January 24, 2018 at 16:28:42, Nick Allen (n...@nickallen.org) wrote:
>>>>>
>>>>> +1 to a standard prefix for all Metron indices. I've had the same
>>>> thought
>>>>> myself and you laid out the advantages well.
>>>>>
>>>>> On Wed, Jan 24, 2018 at 3:47 PM zeo...@gmail.com <zeo...@gmail.com>
>>>> wrote:
>>>>>> I agree with having a metron_ prefix for ES indexes, and the timing.
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> On Wed, Jan 24, 2018 at 3:20 PM Michael Miklavcic <
>>>>>> michael.miklav...@gmail.com> wrote:
>>>>>>
>>>>>>> With the completion of https://github.com/apache/metron/pull/840
>>>>>>> (METRON-939: Upgrade ElasticSearch and Kibana), we have the makings
>>>> for
>>>>> a
>>>>>>> major release rev of Metron in the upcoming release (currently
>>>> slotted
>>>>> to
>>>>>>> 0.4.3, I believe). Since there are non-backwards compatible changes
>>>>>>> pertaining to ES indexing, it seems like a good opportunity to
>>>> revisit
>>>>>> our
>>>>>>> index naming standards.
>>>>>>>
>>>>>>> I propose we add a simple prefix "metron_" to all Metron indexes.
>>>> There
>>>>>> are
>>>>>>> numerous reasons for doing so
>>>>>>>
>>>>>>> - removes the likelihood of index name collisions when we perform
>>>>>>> operations on index wildcard names, e.g. "enrichment_*, indexing_*,
>>>>>>> etc.".
>>>>>>> - ie, this allows us to be more friendly in a multi-tenant ES
>>>>>>> environment for relatively low engineering cost.
>>>>>>> - simplifies the Kibana dashboard a bit. We currently needed to
>>>>>> create a
>>>>>>> special index pattern in order to accommodate multi-index pattern
>>>>>>> matching
>>>>>>> across all metron-specific indexes. Using metron_* would be much
>>>>>> simpler
>>>>>>> and less prone to error.
>>>>>>> - easier for customers to debug and identify Metron-specific indexes
>>>>>> and
>>>>>>> associated data
>>>>>>>
>>>>>>> The reason for making these changes now is that we already have
>>>>> breaking
>>>>>>> changes with ES. Leveraging existing indexed data rather than
>>>> deleting
>>>>>>> indexes and starting from scractch already requires a
>>>>>> re-indexing/migration
>>>>>>> step, so there is no additional effort on the part of users if they
>>>>>> choose
>>>>>>> to attempt a migration. It further makes sense with our current work
>>>>>>> towards upgrading Solr.
>>>>>>>
>>>>>>> We already have a battery of integration and manual tests after the
>>>> ES
>>>>>>> upgrade work that can be leveraged to validate the changes.
>>>>>>>
>>>>>>> Mike Miklavcic
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Jon
>>>>
>>>> --
>>>> A.Nazemian
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
Re: Metron User Community Meeting Call
tRmKpr9adMUN0qfcwJfnmWAQuHY9inQHsSRow=1J5p3hWBZj3Fc4Xy-CytnTi_kafYqRMsY-Ntvr5HlHw=d7cvqZL6hK21y2Y3YW0B49AlEgsICM0D9An4huvIsUI=> >> > >> >> > >> Ahmed would like to talk to the community about >> > >> >> > >> - >> > >> >> > >> Who the GCR group is >> > >> - >> > >> >> > >> How they use Metron 0.4.1 >> > >> - >> > >> >> > >> Walk through their dashboards, UI management screen, nifi >> > >> - >> > >> >> > >> Challenges we faced up until now >> > >> >> > >> I would like to thank Ahmed for stepping forward for this meeting. >> > >> >> > >> If you have something you would like to present or talk about please >> > >> reply here! Maybe we can have people ask for “A better explanation of >> > >> feature X” type things? >> > >> Metron User Community Meetings >> > >> >> > >> User Community Meetings are a means for realtime discussion of >> > >> experiences with Apache Metron, or demonstration of how the community >> is >> > >> using or will be using Apache Metron. >> > >> >> > >> These meetings are geared towards: >> > >> >> > >> - >> > >> >> > >> Demonstrations and knowledge sharing as opposed to technical >> > >> discussion or implementation details from members of the Apache >> > Metron >> > >> Community >> > >> - >> > >> >> > >> Existing Feature demonstrations >> > >> - >> > >> >> > >> Proposed Feature demonstrations >> > >> - >> > >> >> > >> Community feedback >> > >> >> > >> These meetings are *not* for : >> > >> >> > >> - >> > >> >> > >> Support discussions. Those are best left to the mailing lists. >> > >> - >> > >> >> > >> Development discussions. There is another type of meeting for that. >> > >> >> > >> >> > >> >> > >> >> > > >> > > -- >> > > >> > > Jon >> > > >> > >> >> -- >> Thanks, >> Andrew >> >> Subscribe to my book: Streaming Data <http://manning.com/psaltis >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__manning.com_psaltis=DwMGaQ=H50I6Bh8SW87d_bXfZP_8g=yeB_CytRmKpr9adMUN0qfcwJfnmWAQuHY9inQHsSRow=1J5p3hWBZj3Fc4Xy-CytnTi_kafYqRMsY-Ntvr5HlHw=0bpm_zlFmlsG6c8Syr9cEsdZrkKhIuV1mwuJypUBIls=>> >> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306 >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_pub_andrew-2Dpsaltis_1_17b_306=DwMGaQ=H50I6Bh8SW87d_bXfZP_8g=yeB_CytRmKpr9adMUN0qfcwJfnmWAQuHY9inQHsSRow=1J5p3hWBZj3Fc4Xy-CytnTi_kafYqRMsY-Ntvr5HlHw=pRAxEAoEHPf7qW3ly5Ye1Cbo2nvjGlUlGx1UBbcRPhs=>> >> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_intent_user-3Fscreen-5Fname-3Ditmdata=DwMGaQ=H50I6Bh8SW87d_bXfZP_8g=yeB_CytRmKpr9adMUN0qfcwJfnmWAQuHY9inQHsSRow=1J5p3hWBZj3Fc4Xy-CytnTi_kafYqRMsY-Ntvr5HlHw=8ckuW3QNgrz1rYI6eu3yiH09eLd-Msdwyk7CJ13wWMU=>> --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: Metron Alert UI and zero-down time Elasticsearch re-index
Hi Ali, I am not sure I understand what you are trying to do. Are you trying to change the name on the old index, add it to the alias, and then re-index and give the new index the name of the old index? 01.01.2018, 22:30, "Ali Nazemian" <alinazem...@gmail.com>: > Hi All, > > We are using an older version of Metron Alert-UI (Received in Oct 2017) > which sends search queries to ES directly without using Metron Rest API. We > wanted to run a zero-downtime ES reindex process by using ES aliasing. > However, I am not sure how it will impact the search part of Alert-UI > because we need to change it to refer to the alias instead of the old index > name. Please advise how it can be covered in the older version of Metron > Alert-UI. > > Regards, > Ali --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: [DISCUSS] Generating and Interacting with serialized summary objects
the flatfile >>> >>>> - SQL_TRANSFORM(dataframe, spark sql statement): Transforms the >>> >>>> dataframe >>> >>>> - SUMMARIZE(state_init, state_update, state_merge): Summarize the >>> >>>> dataframe using the lambda functions: >>> >>>> - state_init - executed once per worker to initialize the state >>> >>>> - state_update - executed once per row >>> >>>> - state_merge - Merge the worker states into one worker state >>> >>>> - OBJECT_SAVE(obj, output_path) : Save the object obj to the path >>> >>>> output_path on HDFS. >>> >>>> >>> >>>> This would enable more flexibility and composibility than the >>> >>>> configuration-based approach that we have in the flatfile loader. >>> >>>> My concern with this approach, and the reason I didn't do it >>> initially, >>> >>> was >>> >>>> that I think that users will want at least 2 ways to summarize data >>> (or >>> >>>> load data): >>> >>>> >>> >>>> - A configuration based approach, which enables a UI >>> >>>> - A set of stellar functions via the scriptable REPL >>> >>>> >>> >>>> I would argue that both have a place and I started with the >>> >> configuration >>> >>>> based approach as it was a more natural extension of what we already >>> >> had. >>> >>>> I'd love to hear thoughts about this idea too. >>> >>>> >>> >>>> >>> >>>> On Sun, Dec 24, 2017 at 8:20 PM, Casey Stella <ceste...@gmail.com> >>> >>> wrote: >>> >>>> >>> >>>>> Hi all, >>> >>>>> >>> >>>>> I wanted to get some feedback on a sensible plan for something. It >>> >>>>> occurred to me the other day when considering the use-case of >>> >> detecting >>> >>>>> typosquatted domains, that one approach was to generate the set of >>> >>>>> typosquatted domains for some set of reference domains and compare >>> >>>> domains >>> >>>>> as they flow through. >>> >>>>> >>> >>>>> One way we could do this would be to generate this data and import >>> >> the >>> >>>>> typosquatted domains into HBase. I thought, however, that another >>> >>>> approach >>> >>>>> which may trade-off accuracy to remove the network hop and potential >>> >>> disk >>> >>>>> seek by constructing a bloom filter that includes the set of >>> >>> typosquatted >>> >>>>> domains. >>> >>>>> >>> >>>>> The challenge was that we don't have a way to do this currently. We >>> >>> do, >>> >>>>> however, have a loading infrastructure (e.g. the flatfile_loader) >>> and >>> >>>>> configuration (see https://github.com/apache/ >>> >>> metron/tree/master/metron- >>> >>>>> platform/metron-data-management#common-extractor-properties) which >>> >>>>> handles: >>> >>>>> >>> >>>>> - parsing flat files >>> >>>>> - transforming the rows >>> >>>>> - filtering the rows >>> >>>>> >>> >>>>> To enable the new use-case of generating a summary object (e.g. a >>> >> bloom >>> >>>>> filter), in METRON-1378 (https://github.com/apache/metron/pull/879) >>> >> I >>> >>>>> propose that we create a new utility that uses the same extractor >>> >>> config >>> >>>>> add the ability to: >>> >>>>> >>> >>>>> - initialize a state object >>> >>>>> - update the object for every row >>> >>>>> - merge the state objects (in the case of multiple threads, in the >>> >>>>> case of one thread it's not needed). >>> >>>>> >>> >>>>> I think this is a sensible decision because: >>> >>>>> >>> >>>>> - It's a minimal movement from the flat file loader >>> >>>>> - Uses the same configs >>> >>>>> - Abstracts and reuses the existing infrastructure >>> >>>>> - Having one extractor config means that it should be easier to >>> >>>>> generate a UI around this to simplify the experience >>> >>>>> >>> >>>>> All that being said, our extractor config is..shall we >>> say...daunting >>> >>> :). >>> >>>>> I am sensitive to the fact that this adds to an existing difficult >>> >>>> config. >>> >>>>> I propose that this is an initial step forward to support the >>> >> use-case >>> >>>> and >>> >>>>> we can enable something more composable going forward. My concern >>> in >>> >>>>> considering this as the first step was that it felt that the >>> >> composable >>> >>>>> units for data transformation and manipulation suddenly takes us >>> >> into a >>> >>>>> place where Stellar starts to look like Pig or Spark RDD API. I >>> >> wasn't >>> >>>>> ready for that without a lot more discussion. >>> >>>>> >>> >>>>> To summarize, what I'd like to get from the community is, after >>> >>> reviewing >>> >>>>> the entire use-case at https://github.com/cestella/ >>> >>>> incubator-metron/tree/ >>> >>>>> typosquat_merge/use-cases/typosquat_detection: >>> >>>>> >>> >>>>> - Is this so confusing that it does not belong in Metron even as a >>> >>>>> first-step? >>> >>>>> - Is there a way to extend the extractor config in a less >>> >> confusing >>> >>>>> way to enable this? >>> >>>>> >>> >>>>> I apologize for making the discuss thread *after* the JIRAs, but I >>> >> felt >>> >>>>> this one might bear having some working code to consider. >>> >>>>> >>> >>>> >>> >>> >>> >> --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: Metron - Emailing Alerts
I think there may be gaps in doing it with the profiler. You can record stats and counts of different alert types, and maybe even alert ids, but you can't cross-correlate these IDs to the alert body. At least not in the profiler. I was thinking about emailing something that looks like a zeppelin report. You would run it in a cron, export to PDF, and send that out as a summary. It can be a simple list of alerts that match your rule, or it can have aggregations, graphics, metrics, KPI screens, etc. That would be the feature that I would want to discuss and flesh out Thanks, James 13.12.2017, 14:26, "Simon Elliston Ball" <si...@simonellistonball.com>: > We can already do that with profiles I would have thought. Create a profile > that only picks alerts and then base your emails only from the alert events > produced by that profile. Would that create the right batching mechanism (at > a cost of possible higher latency than you might get with a more specific > alert batcher?) > > Simon > >> On 13 Dec 2017, at 21:23, James Sirota <jsir...@apache.org> wrote: >> >> I agree with Simon. If you email each alert individually you will be >> overwhelmed. I think a better idea would be to email alert summaries >> periodically, which is more manageable. This is probably a feature worthy of >> consideration for Metron. >> >> 13.12.2017, 12:19, "Simon Elliston Ball" <si...@simonellistonball.com>: >>> Metron generates alerts onto a Kafka queue, which can be used to integrate >>> with Alert management tools, usually some sort of existing alert >>> aggregation tool. >>> >>> An alternative approach common with this is to have a tool like Apache >>> NiFi attach to the Metron alert feed and send email. >>> >>> The solution here would be to have Metron generate alerts (by adding the >>> is_alert: true flag in the enrichment process) and possibly other flags >>> like alert_email for example, and then have NiFi use ConsumeKafka and then >>> filter out the alert only messages in NiFi to use the PutEmail processor >>> (probably with a ControlRate before it too). >>> >>> Something I would caution is that email is not a great way to manage or >>> send alerts at the volume likely to occur in network monitoring tools. A >>> spike in network traffic can lead to a very large number of emails, which >>> tends to then cause you bigger problems. As such we usually find people >>> want some sort of buffering or aggregation of alerts, hence the use of a an >>> alert management or ticketing solution in front. >>> >>> Simon >>> >>>> On 13 Dec 2017, at 19:06, Ahmed Shah <ahmeds...@cmail.carleton.ca> wrote: >>>> >>>> Hello, >>>> Just wondering if Metron has a feature to email alerts based on rules >>>> that a user defines. >>>> >>>> Example: >>>> Rule A: Email the user 1...@1.com whenever ip_src_addr=100.2.10.* >>>> Rule B: Email the user 1...@1.com whenever payload contains "critical" >>>> >>>> If not, does anyone have any recommendations on where to code these >>>> rules in the Metron stack that uses attributes from the GROK parser? >>>> >>>> -Ahmed >>>> ___ >>>> Ahmed Shah (PMP, M. Eng.) >>>> Cybersecurity Analyst & Developer >>>> GCR - Cybersecurity Operations Center >>>> Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php> >> >> --- >> Thank you, >> >> James Sirota >> PMC- Apache Metron >> jsirota AT apache DOT org --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: [DISCUSS] Community Meetings
I can set up a dedicated Zoom room with a recurrent meeting and give PMC members rights to the room. I think hosting these meetings should not be a problem. I would vote not to record them, but rather provide the notes after the meeting. It's a lot easier to skim through the notes than jump around in a recording. As Simon mentioned, I would also make it explicitly clear that the meetings are dev meetings. These are not user Q and are not meant to be overviews of how different features of Metron work. If we want to do feature demos or provide user content I would want that to be in its own separate meeting. Thanks, James 13.12.2017, 05:00, "Otto Fowler" <ottobackwa...@gmail.com>: > I am ok with just notes and no recording. > > On December 13, 2017 at 04:37:20, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > Good points Larry, we would need to get consent from everyone on the call > to record to properly comply with regulations in some countries. We would > definitely need someone to step up as note taker. > > Something else to think about is intended audience. Previously we’ve had > meeting like this which have been very detailed Dev@ focussed (which is a > great thing) but have rather alienated participants in User@ land. We need > to make it clear what level we’re talking about to be inclusive. > > Simon > >> On 13 Dec 2017, at 00:44, larry mccay <lmc...@apache.org> wrote: >> >> Not sure about posting the recordings - you will need to check and make >> sure that doesn't violate anything. >> >> Just a friendly reminder... >> It is important that meetings have notes and a summary that is sent out >> describing topics to be decided on the mailing list. >> No decisions can be made in the community meeting itself - this gives >> others in other timezones and commitments review and voice in the > > decisions. >> If it didn't happen on the mailing lists then it didn't happen. :) >> >> On Tue, Dec 12, 2017 at 1:39 PM, Simon Elliston Ball < >> si...@simonellistonball.com> wrote: >> >>> Yes, I do. >>> >>> I suspect the best bet will be to post recordings somewhere on the >>> apache.org <http://apache.org/> metron site. >>> >>> Simon >>> >>>> On 12 Dec 2017, at 18:36, Otto Fowler <ottobackwa...@gmail.com> wrote: >>>> >>>> Excellent, do you have the > 40 min + record option? >>>> >>>> On December 12, 2017 at 13:19:55, Simon Elliston Ball ( >>>> si...@simonellistonball.com) wrote: >>>> >>>> Happy to volunteer a zoom room. That seems to have worked for most in > > the >>>> past. >>>> >>>> Simon >>>> >>>>> On 12 Dec 2017, at 18:09, Otto Fowler <ottobackwa...@gmail.com> wrote: >>>>> >>>>> Thanks! I think I’d like something hosted though. >>>>> >>>>> On December 12, 2017 at 11:18:52, Ahmed Shah ( >>> ahmeds...@cmail.carleton.ca) >>>>> wrote: >>>>> >>>>> Hello, >>>>> >>>>> wrt "- How are we going to host it"... >>>>> >>>>> I've used BigBlueButton as an end user at our University. >>>>> >>>>> It is LGPL open source. >>>>> >>>>> https://bigbluebutton.org/ >>>>> https://bigbluebutton.org/developers/ >>>>> >>>>> -Ahmed >>>>> >>>>> ___ >>>>> Ahmed Shah (PMP, M. Eng.) >>>>> Cybersecurity Analyst & Developer >>>>> GCR - Cybersecurity Operations Center >>>>> Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php> >>>>> >>>>> >>>>> From: Otto Fowler <ottobackwa...@gmail.com> >>>>> Sent: December 11, 2017 4:41 PM >>>>> To: dev@metron.apache.org >>>>> Subject: [DISCUSS] Community Meetings >>>>> >>>>> I think that we all want to have regular community meetings. We may be >>>>> better able to keep to a regular schedule with these meetings if we >>>> spread >>>>> out the responsibility for them from James and Casey, both of whom > > have >>> a >>>>> lot on their plate already. >>>>> >>>>> I would be willing to coordinate and run the meetings, and would > > welcome >>>>> anyone else who wants to help when they can. >>>>> >>>>> The only issue for me is I do not have a web-ex account that I can use >>> to >>>>> hold the meeting. So I’ll need some recommendations for a suitable >>>>> alternative. I have not been able to find an Apache Friendly >>> alternative, >>>>> in the same way that Atlassian is apache friendly. >>>>> >>>>> So - from what I can see we need to: >>>>> >>>>> - Talk through who is going to do it >>>>> - How are we going to host it >>>>> - When are we going to do it >>>>> >>>>> Anything else? >>>>> >>>>> ottO --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: Metron - Emailing Alerts
I agree with Simon. If you email each alert individually you will be overwhelmed. I think a better idea would be to email alert summaries periodically, which is more manageable. This is probably a feature worthy of consideration for Metron. 13.12.2017, 12:19, "Simon Elliston Ball" <si...@simonellistonball.com>: > Metron generates alerts onto a Kafka queue, which can be used to integrate > with Alert management tools, usually some sort of existing alert aggregation > tool. > > An alternative approach common with this is to have a tool like Apache NiFi > attach to the Metron alert feed and send email. > > The solution here would be to have Metron generate alerts (by adding the > is_alert: true flag in the enrichment process) and possibly other flags like > alert_email for example, and then have NiFi use ConsumeKafka and then filter > out the alert only messages in NiFi to use the PutEmail processor (probably > with a ControlRate before it too). > > Something I would caution is that email is not a great way to manage or send > alerts at the volume likely to occur in network monitoring tools. A spike in > network traffic can lead to a very large number of emails, which tends to > then cause you bigger problems. As such we usually find people want some sort > of buffering or aggregation of alerts, hence the use of a an alert management > or ticketing solution in front. > > Simon > >> On 13 Dec 2017, at 19:06, Ahmed Shah <ahmeds...@cmail.carleton.ca> wrote: >> >> Hello, >> Just wondering if Metron has a feature to email alerts based on rules that >> a user defines. >> >> Example: >> Rule A: Email the user 1...@1.com whenever ip_src_addr=100.2.10.* >> Rule B: Email the user 1...@1.com whenever payload contains "critical" >> >> If not, does anyone have any recommendations on where to code these rules >> in the Metron stack that uses attributes from the GROK parser? >> >> -Ahmed >> ___ >> Ahmed Shah (PMP, M. Eng.) >> Cybersecurity Analyst & Developer >> GCR - Cybersecurity Operations Center >> Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php> --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
I agree with Nick. Since the plugin is tightly coupled with Metron why not just pull it into the main repo and version it with the rest of the code? Do we really need the second repo for the plug-in? Thanks, James 16.11.2017, 08:06, "Nick Allen" <n...@nickallen.org>: >> I would suggest that we institute a release procedure for the package >> itself, but I don't think it necessarily has to line up with metron >> releases (happy to be persuaded otherwise). Then we can just link metron >> to metron-bro-plugin-kafka by pointing to specific >> metron-bro-plugin-kafka releases (git tags >> <http://bro-package-manager.readthedocs.io/en/stable/package.html#package- >> versioning> >> ). >> Right now, full-dev spins up against the >> apache/metron-bro-plugin-kafka master branch, which is not a good idea to >> have in place for an upcoming release. That is the crux of why I think we >> need to finalize the move to bro 2.5.2 and the plugin packaging before our >> next release (working on it as we speak). >> Jon > > I replayed Jon's comments from the other thread above. > > My initial thought, is that I would not want to manage two separate release > processes. I don't want to have a roll call, cut release candidates and > test both. > > I was thinking we would just need to change some of the behind-the-scenes > processes handled by the release manager. This is one area where I had > thought using a submodule in Git would help. > > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <n...@nickallen.org> wrote: > >> + Restarting the thread to include mentors. >> >> The code of the 'Kafka Plugin for Bro' is now maintained in the external >> repository that we set up a while back. >> >> - Metron Core: git://git.apache.org/metron.git >> - Kafka Plugin for Bro: git://git.apache.org/ >> metron-bro-plugin-kafka.git >> >> (Q) Do we need to change anything in the release procedure to account for >> this? --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: [DISCUSS] Are/how are you using the ES data pruner?
One thing to keep in mind, as we will be introducing Solr shortly, is to find if something similar to curator exists for Solr. But we'll cross that bridge when we get there 22.11.2017, 22:58, "Ali Nazemian" <alinazem...@gmail.com>: > Sure. I will have a chat internally and come back to you shortly. It was a > quick and dirty work actually just to fix this temporarily. However, it > might be a good starting point. > > On Thu, Nov 23, 2017 at 3:31 PM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> Thanks Ali, that's good feedback. Would you be willing to share any of your >> Curator calls/config and use cases with the community? I'd love to add it >> to a document around ES pruning in the short term, and maybe we could look >> at how to build this into indexing at some point. >> >> Cheers, >> Mike >> >> On Nov 22, 2017 8:53 PM, "Ali Nazemian" <alinazem...@gmail.com> wrote: >> >> > We tried to use it, but we had the same issue. It was not documented. We >> > tried to use it, and we had some issues. It also was not exactly what we >> > wanted, so we decided to create something from scratch by using >> > Elasticsearch Curator. We wanted to have an ability to manage different >> > prune mechanism for different feeds. Having a hard threshold to remove >> > index and Soft threshold to close that index. Maybe it can be a feature >> to >> > add to the indexing JSON config file per feed. >> > >> > Cheers, >> > Ali >> > >> > On Thu, Nov 23, 2017 at 12:20 PM, Michael Miklavcic < >> > michael.miklav...@gmail.com> wrote: >> > >> > > From what I can tell, the data pruner isn't documented anywhere, so I'm >> > > curious if anybody is using this, and if so, how are you using it? >> > > >> > > - >> > > https://github.com/apache/metron/blob/master/metron- >> > > platform/metron-data-management/README.md >> > > - >> > > https://github.com/apache/metron/blob/master/metron- >> > > platform/metron-data-management/src/main/java/org/ >> > > apache/metron/dataloads/bulk/ElasticsearchDataPrunerRunner.java >> > > - >> > > https://github.com/apache/metron/blob/master/metron- >> > > platform/metron-data-management/src/main/java/org/ >> > > apache/metron/dataloads/bulk/DataPruner.java >> > > >> > > It looks to me that it allows you to specify the start date and a >> number >> > of >> > > days for lookback from the start date to purge along with a regex >> pattern >> > > to match the index name. It also does not look like it has any built-in >> > > scheduling semantics, so I assume this was a cron job. I think that >> about >> > > covers it. Anything I've missed? >> > > >> > > I'm adding a quick doc write-up to METRON-939 ( >> > > https://github.com/apache/metron/pull/840) for using Curator to prune >> > > indices from Elasticsearch. It is desirable to make sure I've covered >> > > existing use cases. >> > > >> > > Best, >> > > Mike >> > > >> > >> > >> > >> > -- >> > A.Nazemian >> > > > -- > A.Nazemian --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/811 +1 from me as well. Great job @justinleet ---
[GitHub] metron issue #796: METRON-1224: Add time range selection to search control
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 +1 ---
[GitHub] metron issue #796: METRON-1224: Add time range selection to search control
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 + 1. Gret job. all pass login to application â should display error message for invalid credentials â should login for valid credentials â should logout metron-alerts App â should have all the UI elements â should have all pagination controls and they should be working â should have all settings controls and they should be working â play pause should start polling and stop polling â should select columns from table configuration â should have all time-range controls â should have all time range values populated - 1 â should have all time range values populated - 2 â should have all time range values populated - 3 â should have all time range values populated - 4 â should disable date picker when timestamp is present in search â should have now included when to date is empty â should have all time-range included while searching metron-alerts configure table â should select columns from table configuration â should rename columns from table configuration metron-alerts Search â should display all the default values for saved searches â should have all save search controls and they save search should be working â should populate search items when selected on table â should delete search items from search box â should delete first search items from search box having multiple search fields â manually entering search queries to search box and pressing enter key should search metron-alerts tree view â should have all group by elements â drag and drop should change group order â should have group details for single group by â should have group details for multiple group by â should have sort working for group details for multiple sub groups â should have search working for group details for multiple sub groups metron-alerts facets â should display facets data â should expand all facets â should have all facet values â should collapse all facets metron-alerts alert status â should change alert status for multiple alerts to OPEN â should change alert status for multiple alerts to DISMISS â should change alert status for multiple alerts to ESCALATE â should change alert status for multiple alerts to RESOLVE â should change alert status for multiple alerts to OPEN in tree view metron-alerts alert status â should change alert statuses â should add comments for table view â should add comments for tree view Executed 42 of 42 specs SUCCESS in 5 mins 2 secs. [01:01:00] I/launcher - 0 instance(s) of WebDriver still running [01:01:00] I/launcher - chrome #01 passed ---
[GitHub] metron issue #796: METRON-1224: Add time range selection to search control
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 login to application â should display error message for invalid credentials â should login for valid credentials â should logout metron-alerts App â should have all the UI elements â should have all pagination controls and they should be working â should have all settings controls and they should be working â play pause should start polling and stop polling â should select columns from table configuration â sould have all time-range controls â sould have all time-range included while searching - Expected 'Alerts (0)' to equal 'Alerts (5)'. metron-alerts configure table â should select columns from table configuration metron-alerts Search â should display all the default values for saved searches â should have all save search controls and they save search should be working â should populate search items when selected on table â should delete search items from search box â should delete first search items from search box having multiple search fields â manually entering search queries to search box and pressing enter key should search metron-alerts tree view â should have all group by elements â drag and drop should change group order â should have group details for single group by â should have group details for multiple group by â should have sort working for group details for multiple sub groups â should have search working for group details for multiple sub groups metron-alerts facets â should display facets data â should expand all facets â should have all facet values â should collapse all facets metron-alerts alert status â should change alert status for multiple alerts to OPEN â should change alert status for multiple alerts to DISMISS â should change alert status for multiple alerts to ESCALATE â should change alert status for multiple alerts to RESOLVE â should change alert status for multiple alerts to OPEN in tree view - Failed: element not visible (Session info: chrome=61.0.3163.100) (Driver info: chromedriver=2.33.506106 (8a06c39c4582fbfbab6966dbb1c38a9173bfb1a2),platform=Mac OS X 10.12.6 x86_64) metron-alerts alert status â should change alert statuses â should add comments for table view â should add comments for tree view ** *Failures* ** 1) metron-alerts App sould have all time-range included while searching - Expected 'Alerts (0)' to equal 'Alerts (5)'. 2) metron-alerts alert status should change alert status for multiple alerts to OPEN in tree view - Failed: element not visible (Session info: chrome=61.0.3163.100) (Driver info: chromedriver=2.33.506106 (8a06c39c4582fbfbab6966dbb1c38a9173bfb1a2),platform=Mac OS X 10.12.6 x86_64) ---
[GitHub] metron issue #796: METRON-1224: Add time range selection to search control
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 @iraghumitra looks like everything has been addressed. I am +1 on my side, but lets have @merrimanr chime in ---
[GitHub] metron issue #796: METRON-1224: Add time range selection to search control
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 A few things didn't work for me. First, when I select a time range of (t-x minutes) the start and end time does not fill in per screen shot below. https://user-images.githubusercontent.com/4854764/31926145-b2ac5c52-b841-11e7-9fde-aa9d6ba3267d.png;> The time box was initially populated with the word "now", which did not allow me to save. I had to manually enter valid timestamps in order for this to work: https://user-images.githubusercontent.com/4854764/31926214-0d02a616-b842-11e7-8811-b5dae19f63d4.png;> When I click to save a search I get the following exception: TypeError: path must be absolute or specify root to res.sendFile at ServerResponse.sendFile (/usr/metron/0.4.1/web/expressjs/node_modules/express/lib/response.js:410:11) at /usr/metron/0.4.1/web/expressjs/alerts-server.js:71:7 at Layer.handle [as handle_request] (/usr/metron/0.4.1/web/expressjs/node_modules/express/lib/router/layer.js:95:5) at next (/usr/metron/0.4.1/web/expressjs/node_modules/express/lib/router/route.js:137:13) at Route.dispatch (/usr/metron/0.4.1/web/expressjs/node_modules/express/lib/router/route.js:112:3) at Layer.handle [as handle_request] (/usr/metron/0.4.1/web/expressjs/node_modules/express/lib/router/layer.js:95:5) at /usr/metron/0.4.1/web/expressjs/node_modules/express/lib/router/index.js:281:22 at param (/usr/metron/0.4.1/web/expressjs/node_modules/express/lib/router/index.js:354:14) at param (/usr/metron/0.4.1/web/expressjs/node_modules/express/lib/router/index.js:365:14) at Function.process_params (/usr/metron/0.4.1/web/expressjs/node_modules/express/lib/router/index.js:410:3) ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/811 @nickwallen to avoid scope creep on this PR I created a follow-on PR to figure out how to represent meta alerts in the facet panel. https://issues.apache.org/jira/browse/METRON-1276 I agreee with @merrimanr that meta alerts contains multiple alerts each with with distinct meta data and faceting them with the non-grouped alerts doesn't make sense IMO. We do need to figure out a way to visualize metadata of the meta alert (maybe have a separate facet panel for meta alerts?), but I think this will add scope to this PR and therefore should be resolved outside of this PR. ---
[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/811 @nickwallen what you are looking at is a desired behavior. If the alerts are a part of the meta alert they do not appear in the facets ---
new committer: Raghu Mitra
The Project Management Committee (PMC) for Apache Metron has invited Raghu Mitra to become a committer and we are pleased to announce that he has accepted. Being a committer enables easier contribution to the project since there is no need to go via the patch submission process. This should enable better productivity. Being a PMC member enables assistance with the management and to guide the direction of the project.
[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/803 You should not have empty meta alerts. That does not make sense ---
[GitHub] metron issue #579: METRON-941 fix PaloAltoParser
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/579 Hi we would like to get this into the next release. @ctramnitz we'll be happy to help you fix it ---
[GitHub] metron issue #710: Metron-1083: Add filters using faceted search capabilitie...
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/710 Ok, I opened https://issues.apache.org/jira/browse/METRON-1250 as a follow on jira for this ---
[GitHub] metron issue #796: METRON-1224: Add time range selection to search control
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 I tested the PR. The only issue I see is that when I paste the timestamp or manually type it into the boxes it overwrites it with the calendar entries. So essentially the only way to get the timestamp in is to enter it through the calendar selection. Other than that it works great. I tested the positive case and the negative case. The search functionality works. I am a conditional +1 on this, because I do want to be able to enter the timestamps manually (not through the calendar picker). If you can fix that I think this PR is good to go ---
[GitHub] metron issue #796: METRON-1224: Add time range selection to search control
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 @ottobackwards @iraghumitra i already filed a feature request on that: https://issues.apache.org/jira/browse/METRON-1248 ---
[GitHub] metron pull request #788: METRON-1223: Support for adding comments to alerts
Github user james-sirota commented on a diff in the pull request:
https://github.com/apache/metron/pull/788#discussion_r144174823
--- Diff:
metron-interface/metron-alerts/src/app/alerts/alert-details/alert-details.component.ts
---
@@ -133,6 +173,40 @@ export class AlertDetailsComponent implements OnInit {
});
}
+ onAddComment() {
+let alertComment = new AlertComment(this.alertCommentStr,
this.authenticationService.getCurrentUserName(), new Date().getTime());
+let tAlertComments = this.alertCommentsWrapper.map(alertsWrapper =>
alertsWrapper.alertComment);
+tAlertComments.unshift(alertComment);
+this.patchAlert(new Patch('add', '/comments', tAlertComments));
+ }
+
+ patchAlert(patch: Patch) {
+let patchRequest = new PatchRequest();
+patchRequest.guid = this.alertSource.guid;
+patchRequest.index = this.alertIndex;
+patchRequest.patch = [patch];
+patchRequest.sensorType = this.alertSourceType;
+
+this.updateService.patch(patchRequest).subscribe(() => {
+ this.getData();
+});
+ }
+
+ onDeleteComment(index: number) {
+let commentText = 'Do you wish to delete the comment ';
+if (this.alertCommentsWrapper[index].alertComment.comment.length > 25
) {
+ commentText += ' \'' +
this.alertCommentsWrapper[index].alertComment.comment.substr(0, 25) + '...\'';
+} else {
+ commentText += ' \'' +
this.alertCommentsWrapper[index].alertComment.comment + '\'';
+}
+
+
this.metronDialogBox.showConfirmationMessage(commentText).subscribe(response =>
{
+ if (response) {
+this.alertCommentsWrapper.splice(index, 1);
+this.patchAlert(new Patch('add', '/comments',
this.alertCommentsWrapper.map(alertsWrapper => alertsWrapper.alertComment)));
--- End diff --
What I would recommend is just to document around potential issues. I
think what we have here is fine
---
Re: [DISCUSS] Upgrading Elasticsearch from 2.x to 5.x
t; +
>>> > > > "\"message\":\"trying out Elasticsearch\"" +
>>> > > > "}";
>>> > > > HttpEntity entity = new NStringEntity(jsonString,
>>> > > > ContentType.APPLICATION_JSON);
>>> > > > Response response = restClient.performRequest("PUT",
>>> "/posts/doc/1",
>>> > > > params, entity);
>>> > > >
>>> > > > *High Level*
>>> > > >
>>> > > > IndexRequest indexRequest = new IndexRequest("posts", "doc", "1")
>>> > > > .source("user", "kimchy",
>>> > > > "postDate", new Date(),
>>> > > > "message", "trying out Elasticsearch");
>>> > > >
>>> > > > *Note*: there are a few ways to do this with the high level API, but
>>> > this
>>> > > > was the most concise for me to offer a comparison of benefits over
>>> the
>>> > > low
>>> > > > level API.
>>> > > >
>>> > > > *Thoughts/Recommendations*: I do think we should migrate to the new
>>> > API.
>>> > > I
>>> > > > think the question is which of the new APIs we should use. The high
>>> > level
>>> > > > client seems to shield us from having to deal with constructing
>>> special
>>> > > > JSON handling code, whereas the low level client handles all
>>> versions
>>> > of
>>> > > > ES. I don't have a good feel (yet) for just how much work it would
>>> > > require
>>> > > > to use the low level API, or how difficult it would be to add new
>>> > request
>>> > > > features in the future. Actually, we could probably leverage
>>> existing
>>> > > code
>>> > > > we have for dealing with JSON maps, so this might be really easy.
>>> > Someone
>>> > > > with more experience in Metron's ES client use might have a better
>>> idea
>>> > > of
>>> > > > the pros and cons to this. The high level client appears to handle
>>> > > > everything all JSON manipulation for us, but we lose the benefit of
>>> a
>>> > > > simpler dependency tree and support for all versions of ES. My only
>>> > > concern
>>> > > > with "supports all versions" is that I have to imagine there are
>>> > specific
>>> > > > calls that we'd have to be careful of when constructing the JSON
>>> > > requests,
>>> > > > so it's unclear to me if this is better or worse in the end.
>>> > > >
>>> > > > Best,
>>> > > > Mike
>>> > > >
>>> > > >
>>> > > > 1. https://www.elastic.co/blog/state-of-the-official-
>>> > > > elasticsearch-java-clients
>>> > > > 2. https://www.elastic.co/guide/en/elasticsearch/client/java-
>>> > > > rest/current/java-rest-high-compatibility.html
>>> > > > <https://www.elastic.co/guide/en/elasticsearch/client/java-
>>> > > > rest/current/java-rest-high-compatibility.html>
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Wed, Sep 27, 2017 at 8:03 PM, Michael Miklavcic <
>>> > > > michael.miklav...@gmail.com> wrote:
>>> > > >
>>> > > > > I am working on upgrading Elasticsearch and Kibana. There are
>>> quite a
>>> > > few
>>> > > > > changes involved with this vix. I believe I'm mostly finished with
>>> > the
>>> > > > > Ambari mpack side of things, however we currently only support one
>>> > > > version
>>> > > > > with no backwards compatibility. What is the community's thoughts
>>> on
>>> > > > this?
>>> > > > >
>>> > > > > Here is some work contributed to the community that I'm
>>> referencing
>>> > > while
>>> > > > > working on this upgrade - https://github.com/apache/
>>> > > > metron/pull/619/files
>>> > > > >
>>> > > > > Best,
>>> > > > > Michael Miklavcic
>>> > > > >
>>> > > >
>>> > >
>>> >
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
Re: Metron 0.4.2 release date
I would expect ES 5.x support to be in the next version of Metron 08.10.2017, 18:18, "zeo...@gmail.com" <zeo...@gmail.com>: > There's an ongoing conversation regarding client support in Metron here > <https://lists.apache.org/thread.html/0c5a837c901dd057420dd8c6b673dc33ba88a8d97545d5b58856cfe8@%3Cdev.metron.apache.org%3E> > . > > Jon > > On Sun, Oct 8, 2017 at 9:02 PM Ali Nazemian <alinazem...@gmail.com> wrote: > >> Hi Jon, >> >> For the Elasticsearch, I am looking for the support from the client side >> rather than a full Metron mpack that includes ES 5.x. As long as Metron >> Alert-UI and indexing can support ES 5, I am fine. Is that the scope of >> Metron-939? >> >> Cheers, >> Ali >> >> On Mon, Oct 9, 2017 at 11:04 AM, zeo...@gmail.com <zeo...@gmail.com> >> wrote: >> >> > As of right now I'm not aware of any discussions regarding a next >> release, >> > and I believe the METRON-777 features are at least a few months out from >> > being reviewed and merged in (There is a fair amount of work in chunking >> it >> > up to be reviewed, then work to review and merge it in). ES 5.x is also >> in >> > progress but not even open as a PR yet, let alone in master and ready to >> be >> > included in a release. I'm really looking forward to those >> > changes/improvements as well, but I wouldn't be expecting them in the >> next >> > few months, and trying to look further into the future than that at this >> > point would be difficult. >> > >> > That said, if anybody else has a more detailed timeline in mind, I would >> > love to hear more. >> > >> > Jon >> > >> > On Sun, Oct 8, 2017, 09:05 Ali Nazemian <alinazem...@gmail.com> wrote: >> > >> > > Hi all, >> > > >> > > I was wondering when Metron 0.4.2 will be released and whether it >> > includes >> > > Metron-777 and Elasticsearch 5.x or not? >> > > >> > > Cheers, >> > > Ali >> > > >> > -- >> > >> > Jon >> > >> >> -- >> A.Nazemian > -- > > Jon --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
Re: who is having problems installing?
Ok. In that case instead of having a meeting I'll clean up the docs. I'll try to finish this up by the end of the week. Thanks, James 06.10.2017, 13:44, "zeo...@gmail.com" <zeo...@gmail.com>: > First would be to migrate docs from the wiki into the site-book so we have > a more concise place to point people to regarding documentation, because > there is some good stuff in the wiki, and some good things in the > site-books, but attempts to link them together is currently broken all over > the place (perfect example - this > <https://cwiki.apache.org/confluence/display/METRON/Metron+Architecture> is > the architecture page on the wiki). I am thinking about some more concise > areas. An example would cover things like "an intro to the environment" > (Maybe an architecture sub-bullet?) and goes through Storm, Kafka, HDFS, > etc. and covers "How its used" in a more succinct way. Ignoring that, I > was thinking about things like: > > https://cwiki.apache.org/confluence/display/METRON/Data+Loads > https://cwiki.apache.org/confluence/display/METRON/UI > https://cwiki.apache.org/confluence/display/METRON/Deployment+Scripts > > Like I said before, some of this is covered in one location > <https://metron.apache.org/current-book/metron-interface/metron-alerts/index.html>, > but not another <https://cwiki.apache.org/confluence/display/METRON/UI>, > which is extremely confusing. If I found a place where documentation said > it was coming soon, I wouldn't necessarily keep digging to find > documentation for that item. > > Another thing I noticed via my recent perusal was that > https://cwiki.apache.org/confluence/display/METRON/Streaming is not > accurate. Missing indexing, errors, etc. I'm sure there are plenty more > examples as well, and I don't think it's reasonable to point people to the > wiki almost at all any longer (the squid walk-through is a good example of > something still very valuable) because doing so is overly confusing/nuanced > - we really need to get it migrated to the site-book. > > Jon > > On Fri, Oct 6, 2017 at 2:22 PM James Sirota <jsir...@apache.org> wrote: > >> Can you give an example? My personal view is that our docs explain Metron >> fundamentals pretty well. If this is not the case, then would be willing >> to take a look and see how we can make them more consumable. The problem >> with videos is that they become out of date very quickly and it's a lot of >> effort to re-record them. >> >> Thanks, >> James >> >> 06.10.2017, 11:05, "zeo...@gmail.com" <zeo...@gmail.com>: >> > To generalize a bit, I think it would be helpful to have a single or >> series >> > of recordings, write-ups, or even just pointers to some good high-level >> > docs to introduce people to each component used in Metron, and then a >> > description of how it's used in the Metron environment. I know I spend a >> > lot of time talking with people hitting the fundamentals, which is to be >> > expected for a growing project like ours. That helps provide a super >> > high-level context for what is happening in the cluster and where to look >> > if you're seeing certain types of issues. >> > >> > Jon >> > >> > On Fri, Oct 6, 2017 at 1:56 PM James Sirota <jsir...@apache.org> wrote: >> > >> >> Hi Guys, >> >> >> >> How about a meeting at 11 AM PST on this? Can everyone who needs to >> make >> >> the meeting? If you could come with a Hadoop cluster (including Kafka, >> >> storm, HDFS, Hbase) pre-installed I can walk you through the steps >> required >> >> to install Metron. Does that seem reasonable? >> >> >> >> Thanks, >> >> James >> >> >> >> 04.10.2017, 23:01, "Ronirose Caryll De Castro" < >> >> ronirose.decas...@pointwest.com.ph>: >> >> > That's wonderful and very generous of the team. >> >> > >> >> > Here's on my end: >> >> > >> >> > - What environment are you installing on (a single VM, multiple VMs, >> >> bare >> >> > metal, AWS, etc) >> >> > == Single VM, but I'm trying to install on multiple VMs >> >> > - What OS are you using >> >> > == Ubuntu Xenial and Zesty, but trying to use CentOS >> >> > - How many sensors are you going to be consuming >> >> > == Unidentified. I was doing a test install at the moment. >> >> > >> >> > *Tha
Re: SUM aggregator not working?
lt:splitter:end:ts": "1507143907148",
>>> "threat:triage:rules:0:score": 20,
>>> "timestamp": 1507143907108,
>>> "threat:triage:rules:0:reason": "208.110.73.106 is not an WORK
>>> network!",
>>> "awsRegion": "us-east-1",
>>> "is_work": false,
>>> "userIdentity:userName": "",
>>> "enrichmentsplitterbolt:splitter:end:ts": "1507143907143",
>>> "threat:triage:score": 20,
>>> "is_alert": "true",
>>> "userAgent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0)
>>> Gecko/20100101 Firefox/56.0",
>>> "adapter:stellaradapter:begin:ts": "1507143907145",
>>> "eventType": "AwsConsoleSignIn",
>>> "userIdentity:arn": "arn:aws:iam:::user/",
>>> "userIdentity:accountId": "",
>>> "userIdentity:type": "IAMUser",
>>> "threatintelsplitterbolt:splitter:begin:ts": "1507143907148",
>>> "guid": "95617686-bd39-46ff-b5c0-db3aeb5b6bab",
>>> "additionalEventData:LoginTo": "https://console.aws.amazon.co
>>> m/console/home?state=hashArgs%23=true",
>>> "responseElements:ConsoleLogin": "Success"
>>> },
>>> "fields": {
>>> "adapter:stellaradapter:end:ts": [
>>> 1507143907145
>>> ],
>>> "threatinteljoinbolt:joiner:ts": [
>>> 1507143907153
>>> ],
>>> "enrichmentsplitterbolt:splitter:end:ts": [
>>> 1507143907143
>>> ],
>>> "enrichmentsplitterbolt:splitter:begin:ts": [
>>> 1507143907143
>>> ],
>>> "enrichmentjoinbolt:joiner:ts": [
>>> 1507143907147
>>> ],
>>> "adapter:stellaradapter:begin:ts": [
>>> 1507143907145
>>> ],
>>> "eventTime": [
>>> 1507143451000
>>> ],
>>> "threatintelsplitterbolt:splitter:begin:ts": [
>>> 1507143907148
>>> ],
>>> "threatintelsplitterbolt:splitter:end:ts": [
>>> 1507143907148
>>> ],
>>> "timestamp": [
>>> 1507143907108
>>> ]
>>> },
>>> "sort": [
>>> 1507143451000
>>> ]
>>> }
>>>
>>> This is my sensor configuration:
>>>
>>> {
>>> "enrichment": {
>>> "fieldMap": {
>>> "stellar": {
>>> "config": {
>>> "is_work": "IN_SUBNET(if
>>> IS_IP(sourceIPAddress) then sourceIPAddress else NULL, '1.2.3.4/16', '
>>> 5.6.7.8/23')"
>>> }
>>> }
>>> },
>>> "fieldToTypeMap": {},
>>> "config": {}
>>> },
>>> "threatIntel": {
>>> "fieldMap": {
>>> "stellar": {
>>> "config": [
>>> "is_alert := exists(is_work)
>>> &&
>>> is_work != true && eventName == \"ConsoleLogin\"",
>>> "is_alert := is_alert ||
>>> (eventName == \"ConsoleLogin\" &&
>>> userIdentity:sessionContext:attributes:mfaAuthenticated
>>> == \"False\")",
>>> "is_alert := is_alert ||
>>> (eventName == \"ConsoleLogin\" && additionalEventData:MFAUsed ==
>>> \"No\")"
>>> ]
>>> }
>>> },
>>> "fieldToTypeMap": {},
>>> "config": {},
>>> "triageConfig": {
>>> "riskLevelRules": [
>>> {
>>> "name": "Not WORK",
>>> "comment": "Checks whether the
>>> field is_work is true or false.",
>>> "rule": "is_work == false",
>>> "score": 20,
>>> "reason": "FORMAT('%s is not
>>> an
>>> WORK network!', sourceIPAddress)"
>>> },
>>> {
>>> "name": "MFA",
>>> "comment": "Checks whether MFA
>>> used or not.",
>>> "rule":
>>> "userIdentity:sessionContext:attributes:mfaAuthenticated == 'False'",
>>> "score": 20,
>>> "reason": null
>>> },
>>> {
>>> "name": "MFA2",
>>> "comment": "Checks whether MFA
>>> used or not.",
>>> "rule":
>>> "additionalEventData:MFAUsed == 'No'",
>>> "score": 20,
>>> "reason": null
>>> }
>>> ],
>>> "aggregator": "SUM",
>>> "aggregationConfig": {}
>>> }
>>> },
>>> "configuration": {}
>>> }
>>>
>>> Any idea why the score isn't 40? I would expect riskLevelRule 1 & 2 to
>>> be
>>> SUMmed?
---
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: who is having problems installing?
Can you give an example? My personal view is that our docs explain Metron fundamentals pretty well. If this is not the case, then would be willing to take a look and see how we can make them more consumable. The problem with videos is that they become out of date very quickly and it's a lot of effort to re-record them. Thanks, James 06.10.2017, 11:05, "zeo...@gmail.com" <zeo...@gmail.com>: > To generalize a bit, I think it would be helpful to have a single or series > of recordings, write-ups, or even just pointers to some good high-level > docs to introduce people to each component used in Metron, and then a > description of how it's used in the Metron environment. I know I spend a > lot of time talking with people hitting the fundamentals, which is to be > expected for a growing project like ours. That helps provide a super > high-level context for what is happening in the cluster and where to look > if you're seeing certain types of issues. > > Jon > > On Fri, Oct 6, 2017 at 1:56 PM James Sirota <jsir...@apache.org> wrote: > >> Hi Guys, >> >> How about a meeting at 11 AM PST on this? Can everyone who needs to make >> the meeting? If you could come with a Hadoop cluster (including Kafka, >> storm, HDFS, Hbase) pre-installed I can walk you through the steps required >> to install Metron. Does that seem reasonable? >> >> Thanks, >> James >> >> 04.10.2017, 23:01, "Ronirose Caryll De Castro" < >> ronirose.decas...@pointwest.com.ph>: >> > That's wonderful and very generous of the team. >> > >> > Here's on my end: >> > >> > - What environment are you installing on (a single VM, multiple VMs, >> bare >> > metal, AWS, etc) >> > == Single VM, but I'm trying to install on multiple VMs >> > - What OS are you using >> > == Ubuntu Xenial and Zesty, but trying to use CentOS >> > - How many sensors are you going to be consuming >> > == Unidentified. I was doing a test install at the moment. >> > >> > *Thank you!* >> > *Caryll* >> > >> > On Thu, Oct 5, 2017 at 4:01 AM, James Sirota <jsir...@apache.org> wrote: >> > >> >> Yes the intent is for everyone that has any type of metron installation >> >> issue or question attend the meeting >> >> >> >> 03.10.2017, 17:35, "Ronirose Caryll De Castro" < >> >> ronirose.decas...@pointwest.com.ph>: >> >> > Can those who are planning to install Metron join the meeting? >> >> > >> >> > *Thank you!* >> >> > *Caryll* >> >> > >> >> > On Wed, Oct 4, 2017 at 7:11 AM, James Sirota <jsir...@apache.org> >> wrote: >> >> > >> >> >> Hi Guys, >> >> >> >> >> >> How many people do we have with questions about installing Metron? I >> >> can >> >> >> take some time later in the week to schedule a meeting and get >> everyone >> >> >> unstuck >> >> >> >> >> >> --- >> >> >> Thank you, >> >> >> >> >> >> James Sirota >> >> >> PMC- Apache Metron >> >> >> jsirota AT apache DOT org >> >> > >> >> > -- >> >> --- >> >> Thank you, >> >> >> >> James Sirota >> >> PPMC- Apache Metron (Incubating) >> >> jsirota AT apache DOT org >> > >> > -- >> > CONFIDENTIALITY NOTICE: This email may contain confidential and >> privileged >> > material for the sole use of the intended recipient(s). Any review, use, >> > distribution or disclosure by others is strictly prohibited. If you have >> > received this communication in error, please notify the sender >> immediately >> > by e-mail and delete the message and any file attachments from your >> > computer. There is no warranty that this email is error, virus or defect >> > free. If this is a private communication it does not represent the views >> of >> > Pointwest Technologies Corporation or their related entities. >> >> --- >> Thank you, >> >> James Sirota >> PPMC- Apache Metron (Incubating) >> jsirota AT apache DOT org > -- > > Jon --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
Re: Cloudtrail use case
I agree. That's the right place to put them 06.10.2017, 06:26, "Casey Stella" <ceste...@gmail.com>: > There is actually a use-cases top level directory with worked examples in > them. They get picked up by the doc book too! I'd suggest putting it > there, thoughts? > > On Fri, Oct 6, 2017 at 8:44 AM, Nick Allen <n...@nickallen.org> wrote: > >> Yes, agreed, Justin. I guess my main point to Laurens was meant to be that >> the actual destination of the use case should be the least of our worries. >> However Laurens wants to write it up will work. If you type it up, throw it >> in an envelope, seal it with a stamp, and physically mail it to me, I will >> make sure it lands in the right place. :) >> >> On Thu, Oct 5, 2017 at 9:20 PM Justin Leet <justinjl...@gmail.com> wrote: >> >> > I know we've had discussions about migrating stuff into docs before. It >> > might be worth resurrecting a more use case focused version of that, >> > instead of starting on the wiki. I assume the end goal is availability >> in >> > the site-book, so even if it's not in a perfect place, I'd rather the >> > effort be spent on making it pretty there. >> > >> > I think there's a few floating around that could use a home, so the >> > discussion might make life easier for multiple things. Some from the >> wiki, >> > some from random READMEs we could relocate and link, some from >> > presentations and so on. >> > >> > Having said all that, I know discuss threads can take a few days to >> > resolve, so wiki and then convert might be the lesser of two evils. >> > >> > >> > On Thu, Oct 5, 2017 at 6:54 PM, Nick Allen <n...@nickallen.org> wrote: >> > >> > > We don't really have a location in the source code for use cases like >> > this >> > > right now. But I think it is so important that we get use cases like >> > this >> > > published somewhere. For now, you could add this to the Wiki. Then >> > later >> > > on we can figure out how to handle that. >> > > >> > > On Thu, Oct 5, 2017 at 6:49 PM, Laurens Vets <laur...@daemon.be> >> wrote: >> > > >> > > > On 2017-10-05 15:45, Laurens Vets wrote: >> > > > >> > > >> Hi, >> > > >> >> > > >> Would anyone be interested in adding a full AWS Cloudtrail use case >> to >> > > >> the Metron documentation? I would roughly consist of: >> > > >> - Apache NiFi configuration to retrieve Cloudtrail logs from S3 and >> > > >> send it to Metron via Kafka. >> > > >> - Complete Metron sensor configuration (enrichment, alerting, >> etc...) >> > > for >> > > >> this. >> > > >> >> > > > >> > > > Sent too soon :( >> > > > >> > > > If anyone would be interested in this documentation, where would add >> > this >> > > > in the source? >> > > > >> > > >> > --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
Re: Need suggestion on how to configure HCP Big Data for Development and Testing
As I mentioned in my previous response, https://community.hortonworks.com/topics/Metron.html is where you want to go for help with Hortonworks products 06.10.2017, 05:34, "Dima Kovalyov" <dima.koval...@sstech.us>: > Hello Ashikin, > > HCP is Hortonworks product and they have installation document here: > https://docs.hortonworks.com/HDPDocuments/HCP1/HCP-1.2.0/bk_installation/content/getting_started.html > Chapter that you are looking for is below: > https://docs.hortonworks.com/HDPDocuments/HCP1/HCP-1.3.0/bk_installation/content/installing_configuring_deploying_hdp_cluster.html > > Dell specific wording: Dell PowerEdge VRTX, M630 and HDD 6006 does not > tell me much about hardware you have. But if I have two guess, I would > suggest to have: > 1 node for Ambari and hdfs, hbase, zookeeper, metrics, zeppelin, spark, > hive, tez and yarn masters. > 1 node for Metron and storm, kafka, metron, es, kibana > 2 nodes + 1 node (ambari one) for all the slaves and masters that > require replication (hdfs datanode, zookeeper, kafka broker, es slaves). > > I have not tried HCP myself, but if I would, I would rather post all my > HCP related questions to HortonWorks forums > (https://community.hortonworks.com/index.html, they have really good > support there) rather then to Apache Metron devs as they are not related > (HortonWorks uses Apache Metron (open-source software) in their HDP > framework). > > - Dima > > On 10/05/2017 04:59 PM, Ashikin Abdullah wrote: >> Hi, can anyone help me to suggest appropriate deployment for Hortonworks >> Cybersecurity Package within this environment. We have Dell PowerEdge VRTX >> with 4 nodes, M630 x 4 and HDD 6006 x 25 (shared storage). >> >> Therefore, how to manage all this resources to properly configured HCP? >> >> Thanks in advance. --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
Re: Configuring HCP Big Data for Development
For questions on the Hortonworks offerings please ask your questions on https://community.hortonworks.com/topics/Metron.html This is a board for Apache Metron. 05.10.2017, 00:57, "Ashikin Abdullah" <ashikin...@gmail.com>: > Hi, can anyone suggest appropriate deployment for Hortonworks Cybersecurity > Package within this environment. We have Dell PowerEdge VRTX with 4 nodes > and 4 HDD M630 (shared storage) x 25. > > Therefore, how to manage all this resources to properly configured HCP? > > Hope you guys can help me. Thanks in advance. ------- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
Re: [DISCUSS] Build broken due to transitive dependencies
Can you run it with the -X flag and paste the error? What version of the gcc
compiler do you have?
02.10.2017, 09:37, "Laurens Vets" <laur...@daemon.be>:
> I might have spoken too soon. This is what I see now on 0.4.1-release:
>
> ...
> [INFO] metron-contrib . SUCCESS [
> 0.006 s]
> [INFO] metron-docker .. SUCCESS [
> 3.088 s]
> [INFO] metron-interface ... SUCCESS [
> 0.057 s]
> [INFO] metron-config .. FAILURE
> [06:54 min]
> [INFO] metron-alerts .. SUCCESS
> [03:44 min]
> [INFO] metron-rest-client . SUCCESS [
> 0.411 s]
> [INFO] metron-rest SUCCESS [
> 26.628 s]
> [INFO] site-book .. SUCCESS [
> 1.136 s]
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 06:56 min (Wall Clock)
> [INFO] Finished at: 2017-10-02T16:33:39+00:00
> [INFO] Final Memory: 240M/3203M
> [INFO]
>
> [ERROR] Failed to execute goal
> com.github.eirslett:frontend-maven-plugin:1.3:npm (ng build) on project
> metron-config: Failed to run task: 'npm run build' failed. (error code
> 1) -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the
> -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :metron-config
>
> On 2017-10-02 08:16, Laurens Vets wrote:
>> I can confirm 0.4.1 (on CentOS 6!) builds for me as well.
>>
>> Are we sure it isn't due to the version of node shipped with the OS?
>>
>> On 2017-10-02 08:04, zeo...@gmail.com wrote:
>>> Hmm, 0.4.1 built fine for me.
>>>
>>> Jon
>>>
>>> On Mon, Oct 2, 2017 at 10:44 AM Casey Stella <ceste...@gmail.com>
>>> wrote:
>>>
>>>> Ok, the build is broken in metron-config due to some transitive
>>>> changes
>>>> that happened in npm-land:
>>>>
>>>> [INFO]
>>>>
>>>>
>>>> /Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32
>>>> [INFO] throw new Error('Cyclic dependency:
>>>> '+JSON.stringify(node))
>>>> [INFO] ^
>>>> [INFO] Error: Cyclic dependency: "[object Object]"
>>>> [INFO] at visit
>>>>
>>>>
>>>> (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:32:13)
>>>> [INFO] at visit
>>>>
>>>>
>>>> (/Users/cstella/Documents/workspace/metron/fork/incubator-metron/metron-interface/metron-config/node_modules/toposort/index.js:48:9)
>>>>
>>>> Evidently one of our transitive dependencies has changed and we have
>>>> ended
>>>> up with a cyclic dependency. I'm not sure where or why yet, but I
>>>> believe
>>>> this breaks both master and our 0.4.1 release (I haven't confirmed
>>>> this
>>>> yet, but I strongly suspect).
>>>>
>>>> While the good work of tracking down this specific error is done, I'd
>>>> like
>>>> to bring up a broader discussion point: our practice of not fixing
>>>> versions
>>>> for our node dependencies. This is, in effect, causing a few
>>>> problems:
>>>>
>>>> - We do not have a consistent, repeatable build.
>>>> - We set ourselves up for possible license violation without
>>>> knowing
>>>> about it (a transitive dependency changes its license)
>>>>
>>>> As we stand, we have a release which doesn't not build after we have
>>>> released it and tested it. It seems to me that we should at a
>>>> minimum as a
>>>> stopgap:
>>>>
>>>> - fix the versions of our dependencies so that they are in a
>>>> working
>>>> state
>>>> - consider a point release to get a working build.
>>>>
>>>> I guess my questions to those of us with more javascript and UI
>>>> experience
>>>> is as follows:
>>>>
>>>> - Does fixing the version of our dependencies actually fix the
>>>> problem
>>>> transitively?
>>>> - IF not, then how do we get a version of a build which is
>>>> consistent
>>>> and repeatable and does not expose us to downstream licensing
>>>> issues?
>>>>
>>>> Thanks,
>>>>
>>>> Casey
---
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: [DISCUSS] Is there a reason for separate Management & Alerts UIs?
At some point in the future we may think about converging them because functions like defining threat rules and setting up profiles may overlap the SOC and ops personnel. But as you said, the initial intent was that the two UIs target two different user personas. 02.10.2017, 11:35, "Nick Allen" <n...@nickallen.org>: > I think the main reason historically is that each UI has different use > cases and user roles. The Management UI will mainly be used by an Security > Platform Engineer, while the Alerts UI will be used by a SOC Analyst, > Investigator or Manager. > > That being said, I am not against a single, unified UI, as long as it is > paired with appropriate role based access controls. > > On Thu, Sep 28, 2017 at 12:10 PM Laurens Vets <laur...@daemon.be> wrote: > >> As the subject says, is there a specific reason to have the Management & >> Alerts UI separate? >> >> Having another option under "Operations" called "Alerts" in the >> Management UI seems to make more sense to me... If it's because they are >> called Management UI and Alerts UI, maybe we should make it more general >> and name it Metron UI? --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
Re: who is having problems installing?
Yes the intent is for everyone that has any type of metron installation issue or question attend the meeting 03.10.2017, 17:35, "Ronirose Caryll De Castro" <ronirose.decas...@pointwest.com.ph>: > Can those who are planning to install Metron join the meeting? > > *Thank you!* > *Caryll* > > On Wed, Oct 4, 2017 at 7:11 AM, James Sirota <jsir...@apache.org> wrote: > >> Hi Guys, >> >> How many people do we have with questions about installing Metron? I can >> take some time later in the week to schedule a meeting and get everyone >> unstuck >> >> --- >> Thank you, >> >> James Sirota >> PMC- Apache Metron >> jsirota AT apache DOT org > > -- > CONFIDENTIALITY NOTICE: This email may contain confidential and privileged > material for the sole use of the intended recipient(s). Any review, use, > distribution or disclosure by others is strictly prohibited. If you have > received this communication in error, please notify the sender immediately > by e-mail and delete the message and any file attachments from your > computer. There is no warranty that this email is error, virus or defect > free. If this is a private communication it does not represent the views of > Pointwest Technologies Corporation or their related entities. --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
Re: who is having problems installing?
Extending this to the user list as well. Whoever needs help can you quickly let me know: - What environment are you installing on (a single VM, multiple VMs, bare metal, AWS, etc) - What OS are you using - How many sensors are you going to be consuming I'll throw a meeting together once I get a feel for what people are doing with the platform. Thanks, James 03.10.2017, 18:22, "Ronirose Caryll De Castro" <ronirose.decas...@pointwest.com.ph>: > Yes, like me I am planning to install Metron and would like to join the > meeting to know what would be the possible issues that I will face and how > to solve them > > *Thank you!* > *Caryll* > > On Wed, Oct 4, 2017 at 9:02 AM, Otto Fowler <ottobackwa...@gmail.com> wrote: > >> Did you mean to send this to users too? >> >> On October 3, 2017 at 19:12:10, James Sirota (jsir...@apache.org) wrote: >> >> Hi Guys, >> >> How many people do we have with questions about installing Metron? I can >> take some time later in the week to schedule a meeting and get everyone >> unstuck >> >> --- >> Thank you, >> >> James Sirota >> PMC- Apache Metron >> jsirota AT apache DOT org > > -- > CONFIDENTIALITY NOTICE: This email may contain confidential and privileged > material for the sole use of the intended recipient(s). Any review, use, > distribution or disclosure by others is strictly prohibited. If you have > received this communication in error, please notify the sender immediately > by e-mail and delete the message and any file attachments from your > computer. There is no warranty that this email is error, virus or defect > free. If this is a private communication it does not represent the views of > Pointwest Technologies Corporation or their related entities. --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
who is having problems installing?
Hi Guys, How many people do we have with questions about installing Metron? I can take some time later in the week to schedule a meeting and get everyone unstuck --- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Link to the recording. You need a WebEx player to view it https://drive.google.com/open?id=0B3a8U3GCzkKNZHR2LUZNcmljcTQ 26.09.2017, 09:20, "Laurens Vets" <laur...@daemon.be>: > 11:30 won't work for me, but that's fine. I only had 1 comment on Otto's > video: What happens when we have 2 parsers/sensors with the same name. > If there's ever a parser/sensor repository, this might be an issue. > > On 2017-09-25 17:38, Otto Fowler wrote: >> 11:30 your time. Sorry I have to pick my kids up from school. 2:30 >> mine. >> >> On September 25, 2017 at 19:41:28, James Sirota (jsir...@apache.org) >> wrote: >> >> Oh sorry, didn't notice that. Otto, when is a good time for you? >> >> 25.09.2017, 16:35, "zeo...@gmail.com" <zeo...@gmail.com>: >>> When is the meeting, given Otto mentioned he can't make 10am? Or did >>> that >>> change >>> >>> Jon >>> >>> On Mon, Sep 25, 2017, 19:19 James Sirota <jsir...@apache.org> wrote: >>> >>>> Great. Thank you, Otto. I would encourage everyone to watch it so >>>> that >> we >>>> have constructive feedback for tomorrow and are able to arrive to a >> decision >>>> Thanks, >>>> James >>>> >>>> 25.09.2017, 08:27, "Otto Fowler" <ottobackwa...@gmail.com>: >>>> > https://youtu.be/-ISycoP3TVA >>>> > >>>> > The video is short and simple. Hopefully it is what you are >>>> looking >> for. >>>> > >>>> > On September 21, 2017 at 16:54:13, zeo...@gmail.com >>>> (zeo...@gmail.com) >> >>>> > wrote: >>>> > >>>> > I won't be able to make it and would really like to make sure >>>> there's >> a >>>> > recording for this one, if possible. I'm unavailable until >>>> Thursday >> of >>>> > next week, but not necessarily suggesting this gets moved. >>>> > >>>> > Jon >>>> > >>>> > On Thu, Sep 21, 2017, 15:04 Otto Fowler <ottobackwa...@gmail.com> >> wrote: >>>> > >>>> >> I can’t make that time, can we make it later in the day? >>>> >> >>>> >> On September 21, 2017 at 11:40:37, James Sirota >>>> (jsir...@apache.org) >>>> >> wrote: >>>> >> >>>> >> https://hortonworks.webex.com/meet/jsirota >>>> > -- >>>> > >>>> > Jon >>>> >>>> --- >>>> Thank you, >>>> >>>> James Sirota >>>> PPMC- Apache Metron (Incubating) >>>> jsirota AT apache DOT org >>> -- >>> >>> Jon >> >> --- >> Thank you, >> >> James Sirota >> PPMC- Apache Metron (Incubating) >> jsirota AT apache DOT org --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Oh sorry, didn't notice that. Otto, when is a good time for you? 25.09.2017, 16:35, "zeo...@gmail.com" <zeo...@gmail.com>: > When is the meeting, given Otto mentioned he can't make 10am? Or did that > change > > Jon > > On Mon, Sep 25, 2017, 19:19 James Sirota <jsir...@apache.org> wrote: > >> Great. Thank you, Otto. I would encourage everyone to watch it so that we >> have constructive feedback for tomorrow and are able to arrive to a decision >> >> Thanks, >> James >> >> 25.09.2017, 08:27, "Otto Fowler" <ottobackwa...@gmail.com>: >> > https://youtu.be/-ISycoP3TVA >> > >> > The video is short and simple. Hopefully it is what you are looking for. >> > >> > On September 21, 2017 at 16:54:13, zeo...@gmail.com (zeo...@gmail.com) >> > wrote: >> > >> > I won't be able to make it and would really like to make sure there's a >> > recording for this one, if possible. I'm unavailable until Thursday of >> > next week, but not necessarily suggesting this gets moved. >> > >> > Jon >> > >> > On Thu, Sep 21, 2017, 15:04 Otto Fowler <ottobackwa...@gmail.com> wrote: >> > >> >> I can’t make that time, can we make it later in the day? >> >> >> >> On September 21, 2017 at 11:40:37, James Sirota (jsir...@apache.org) >> >> wrote: >> >> >> >> https://hortonworks.webex.com/meet/jsirota >> > -- >> > >> > Jon >> >> --- >> Thank you, >> >> James Sirota >> PPMC- Apache Metron (Incubating) >> jsirota AT apache DOT org > -- > > Jon --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
Re: [DISCUSS] Splitting up the Indexing Topology
I have experienced issues with ES and HDFS indexing in production and have previously split out the topologies into two separate topologies. As you state the benefits of this approach are (a) tuning each topology separately, (b) ability to attribute problems to a specific topology (why is something slow?) and (c) graceful degradation. When ES, for example, fails partially or catastrophically and your ES topology goes all kinds of crazy, HDFS topology keeps humming along unaffected. Once Metron-1205 is in you will be able to re-index into ES (or potentially other sources) from HDFS at will. The major con for this architecture is that there is a greater chance that all your data sources will get out of sync because they index/store data at different rates. But even given that I would vote +1 on splitting out the topologies. 25.09.2017, 09:37, "Casey Stella" <ceste...@gmail.com>: > One of the lessons that have bubbled up in doing some performance analysis > is that having the indexing topology share both the ES and the HDFS writer > in the same topology can be problematic from a tuning perspective. > Specifically, it's hard to square that circle and make both perform fast > enough to not cause significant back-pressure in kafka (and often Commit > Exceptions in the kafka spout). > > I wanted to get the community's opinion about the possibility of separating > the two current writers into separate topologies which could be tuned > separately. > > Pros: > > - Practically speaking, tuning separately is often a lot easier than > trying to tune together > - This opens us up with the beginnings of an abstraction that may be > reusable to expose new indexers to Metron > > Cons: > > - It has the potential to mask a problem. We may want to ensure that > the writers write at the same rate and don't get far ahead of one another. > In the current setup, this is inherent in the design. If we separate them, > they may be reading at different rates and one index may get ahead of the > other. > - The management pack section around indexing would need to be > reconsidered if we split them up > > Personally, I'm strongly in favor of splitting them up, but I want to make > sure that we don't miss an important nuance here. The first con is > concerning to me, but I'd argue that another lesson from performance tuning > is that we need to monitor the average partition lag over time in the > management UI for the various consumer groups and ensure that writing keeps > up with reading. If we insist on this assertion being true for all healthy > metron installations, the primary con goes away in my mind. > > Anyway, I'm sure I've missed some pros and cons, so it'd be great to hear > community feedback here. Thoughts? --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Great. Thank you, Otto. I would encourage everyone to watch it so that we have constructive feedback for tomorrow and are able to arrive to a decision Thanks, James 25.09.2017, 08:27, "Otto Fowler" <ottobackwa...@gmail.com>: > https://youtu.be/-ISycoP3TVA > > The video is short and simple. Hopefully it is what you are looking for. > > On September 21, 2017 at 16:54:13, zeo...@gmail.com (zeo...@gmail.com) > wrote: > > I won't be able to make it and would really like to make sure there's a > recording for this one, if possible. I'm unavailable until Thursday of > next week, but not necessarily suggesting this gets moved. > > Jon > > On Thu, Sep 21, 2017, 15:04 Otto Fowler <ottobackwa...@gmail.com> wrote: > >> I can’t make that time, can we make it later in the day? >> >> On September 21, 2017 at 11:40:37, James Sirota (jsir...@apache.org) >> wrote: >> >> https://hortonworks.webex.com/meet/jsirota > -- > > Jon --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
[DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST
Hi Guys, I'd like to propose a community meeting for this Tuesday, Sept.23 and focus on the strategy for getting Metron-777 and related PRs on Otto's feature branch merged into master. Otto, is there a way you can record a youtube of adding a custom via your feature branch by Tueday? I'd like to be able to review that and get everyone's feedback so that we can start formulating our strategy on getting it in. I propose to have this meeting at Tuesday, Sept.23 10AM PST if that works for everyone: https://hortonworks.webex.com/meet/jsirota Call-in toll-free number (US/Canada) 1-877-668-4493 --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
Re: [VOTE] Metron Release Candidate 0.4.1-RC4
v/metron/0.4.1- >> RC4/CHANGES >> > > > >> > > >The github tag to be voted upon is: >> > > >apache-metron-0.4.1-rc4 >> > > > >> > > >The source archive being voted upon can be found here: >> > > > >> > > >> > https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/ >> apache-metron-0.4.1-rc4.tar.gz >> > > > >> > > >The site-book is at: >> > > > >> > > >> > https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/ >> site-book/index.html >> > > > >> > > >Other release files, signatures and digests can be found here: >> > > >https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/ >> > > > >> > > >The release artifacts are signed with the following key: >> > > >4169 AA27 ECB3 1663 in >> > > https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/KEYS >> > > > >> > > >Please vote on releasing this package as Apache Metron 0.4.1 >> > > > >> > > >When voting, please list the actions taken to verify the >> release. >> > > > >> > > >Recommended build validation and verification instructions >> are posted >> > > here: >> > > >https://cwiki.apache.org/confluence/display/METRON/ >> Verifying+Builds >> > > > >> > > >This vote will be open until 5pm PDT Wednesday 13 Sep, due to >> the >> > weekend. >> > > >Thanks, >> > > >--Matt >> > > >(release manager) >> > > > >> > > > >> > > > >> > > >> > -- >> > >> > Jon >> > --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
Re: [ANNOUNCE] Metron community meeting
Hi Guys, apologies about this. I couldn't record yesterday, but Casey posted a synopsis of the meeting. 22.08.2017, 10:27, "James Sirota" <jsir...@apache.org>: > Yes, I will post a recording > > 21.08.2017, 14:57, "Kyle Richardson" <kylerichards...@gmail.com>: >> Unfortunately I have a conflict. Will there be a recording posted afterward >> like previous meetings? >> >> -Kyle >> >> On Mon, Aug 21, 2017 at 10:35 AM Otto Fowler <ottobackwa...@gmail.com> >> wrote: >> >>> Sounds good >>> >>> On August 21, 2017 at 09:43:25, James Sirota (jsir...@apache.org) wrote: >>> >>> Hi Jon, >>> >>> Sure. Lets move it by a day. The reason it's at this time is to give >>> people >>> in India and Europe a chance to attend live. >>> >>> So lets move it to the same time tomorrow >>> >>> I would like to schedule it for Tuesday, Aug. 22. at 10.30AM PST. >>> >>> I would like to have it over webex: >>> >>> >>> https://hortonworks.webex.com/hortonworks/j.php?MTID=m5621bcdbf6163c7df1568b41fc1a1d93 >>> Meeting number: 622 537 955 >>> Meeting password: biFTEuh2 >>> >>> For global callers: >>> >>> >>> https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC=590161912=1 >>> >>> Thanks, >>> James >>> >>> 18.08.2017, 11:02, "zeo...@gmail.com" <zeo...@gmail.com>: >>> > Is it possible to reschedule this to later in the day or another day? >>> That >>> > overlaps with the eclipse on the east cost of the US that some people >>> would >>> > like to enjoy. >>> > >>> > Jon >>> > >>> > On Fri, Aug 18, 2017, 13:48 James Sirota <jsir...@apache.org> wrote: >>> > >>> >> I would like to propose a meeting with the following set of topics: >>> >> >>> >> - What do we want in the next build >>> >> - Where do we want to take the project long-term >>> >> - Any feature requests/comments on existing feature set >>> >> - Feedback on some of the early UI work >>> >> >>> >> Please suggest if there is anything else you want to talk about. >>> >> >>> >> I would like to schedule it for Monday, Aug. 21. at 10.30AM PST. >>> >> >>> >> I would like to have it over webex: >>> >> >>> >> >>> >>> >>> https://hortonworks.webex.com/hortonworks/j.php?MTID=m5621bcdbf6163c7df1568b41fc1a1d93 >>> >> Meeting number: 622 537 955 >>> >> Meeting password: biFTEuh2 >>> >> >>> >> For global callers: >>> >> >>> >> >>> >>> >>> https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC=590161912=1 >>> >> >>> >> Anyone is welcome to join. >>> >> --- >>> >> Thank you, >>> >> >>> >> James Sirota >>> >> PPMC- Apache Metron (Incubated and Hatched) >>> >> jsirota AT apache DOT org >>> > -- >>> > >>> > Jon >>> >>> --- >>> Thank you, >>> >>> James Sirota >>> PPMC- Apache Metron (Incubating) >>> jsirota AT apache DOT org > > --- > Thank you, > > James Sirota > PPMC- Apache Metron (Incubating) > jsirota AT apache DOT org --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
Re: [ANNOUNCE] Metron community meeting
Yes, I will post a recording 21.08.2017, 14:57, "Kyle Richardson" <kylerichards...@gmail.com>: > Unfortunately I have a conflict. Will there be a recording posted afterward > like previous meetings? > > -Kyle > > On Mon, Aug 21, 2017 at 10:35 AM Otto Fowler <ottobackwa...@gmail.com> > wrote: > >> Sounds good >> >> On August 21, 2017 at 09:43:25, James Sirota (jsir...@apache.org) wrote: >> >> Hi Jon, >> >> Sure. Lets move it by a day. The reason it's at this time is to give people >> in India and Europe a chance to attend live. >> >> So lets move it to the same time tomorrow >> >> I would like to schedule it for Tuesday, Aug. 22. at 10.30AM PST. >> >> I would like to have it over webex: >> >> >> https://hortonworks.webex.com/hortonworks/j.php?MTID=m5621bcdbf6163c7df1568b41fc1a1d93 >> Meeting number: 622 537 955 >> Meeting password: biFTEuh2 >> >> For global callers: >> >> >> https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC=590161912=1 >> >> Thanks, >> James >> >> 18.08.2017, 11:02, "zeo...@gmail.com" <zeo...@gmail.com>: >> > Is it possible to reschedule this to later in the day or another day? >> That >> > overlaps with the eclipse on the east cost of the US that some people >> would >> > like to enjoy. >> > >> > Jon >> > >> > On Fri, Aug 18, 2017, 13:48 James Sirota <jsir...@apache.org> wrote: >> > >> >> I would like to propose a meeting with the following set of topics: >> >> >> >> - What do we want in the next build >> >> - Where do we want to take the project long-term >> >> - Any feature requests/comments on existing feature set >> >> - Feedback on some of the early UI work >> >> >> >> Please suggest if there is anything else you want to talk about. >> >> >> >> I would like to schedule it for Monday, Aug. 21. at 10.30AM PST. >> >> >> >> I would like to have it over webex: >> >> >> >> >> >> >> https://hortonworks.webex.com/hortonworks/j.php?MTID=m5621bcdbf6163c7df1568b41fc1a1d93 >> >> Meeting number: 622 537 955 >> >> Meeting password: biFTEuh2 >> >> >> >> For global callers: >> >> >> >> >> >> >> https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC=590161912=1 >> >> >> >> Anyone is welcome to join. >> >> --- >> >> Thank you, >> >> >> >> James Sirota >> >> PPMC- Apache Metron (Incubated and Hatched) >> >> jsirota AT apache DOT org >> > -- >> > >> > Jon >> >> --- >> Thank you, >> >> James Sirota >> PPMC- Apache Metron (Incubating) >> jsirota AT apache DOT org --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
Re: [ANNOUNCE] Metron community meeting
Hi Jon, Sure. Lets move it by a day. The reason it's at this time is to give people in India and Europe a chance to attend live. So lets move it to the same time tomorrow I would like to schedule it for Tuesday, Aug. 22. at 10.30AM PST. I would like to have it over webex: https://hortonworks.webex.com/hortonworks/j.php?MTID=m5621bcdbf6163c7df1568b41fc1a1d93 Meeting number: 622 537 955 Meeting password: biFTEuh2 For global callers: https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC=590161912=1 Thanks, James 18.08.2017, 11:02, "zeo...@gmail.com" <zeo...@gmail.com>: > Is it possible to reschedule this to later in the day or another day? That > overlaps with the eclipse on the east cost of the US that some people would > like to enjoy. > > Jon > > On Fri, Aug 18, 2017, 13:48 James Sirota <jsir...@apache.org> wrote: > >> I would like to propose a meeting with the following set of topics: >> >> - What do we want in the next build >> - Where do we want to take the project long-term >> - Any feature requests/comments on existing feature set >> - Feedback on some of the early UI work >> >> Please suggest if there is anything else you want to talk about. >> >> I would like to schedule it for Monday, Aug. 21. at 10.30AM PST. >> >> I would like to have it over webex: >> >> >> https://hortonworks.webex.com/hortonworks/j.php?MTID=m5621bcdbf6163c7df1568b41fc1a1d93 >> Meeting number: 622 537 955 >> Meeting password: biFTEuh2 >> >> For global callers: >> >> >> https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC=590161912=1 >> >> Anyone is welcome to join. >> --- >> Thank you, >> >> James Sirota >> PPMC- Apache Metron (Incubated and Hatched) >> jsirota AT apache DOT org > -- > > Jon --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
[ANNOUNCE] Metron community meeting
I would like to propose a meeting with the following set of topics: - What do we want in the next build - Where do we want to take the project long-term - Any feature requests/comments on existing feature set - Feedback on some of the early UI work Please suggest if there is anything else you want to talk about. I would like to schedule it for Monday, Aug. 21. at 10.30AM PST. I would like to have it over webex: https://hortonworks.webex.com/hortonworks/j.php?MTID=m5621bcdbf6163c7df1568b41fc1a1d93 Meeting number: 622 537 955 Meeting password: biFTEuh2 For global callers: https://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC=590161912=1 Anyone is welcome to join. --- Thank you, James Sirota PPMC- Apache Metron (Incubated and Hatched) jsirota AT apache DOT org
[GitHub] metron issue #683: METRON-1084: Management UI web server license should be A...
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/683 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #682: modified: NOTICE
GitHub user james-sirota opened a pull request: https://github.com/apache/metron/pull/682 modified: NOTICE ## Contributor Comments [Please place any comments here. A description of the problem/enhancement, how to reproduce the issue, your testing methodology, etc.] ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/james-sirota/metron jsirota/METRON-1081 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/682.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #682 commit aa516a642497db0f815f59fecc9e2cfd87adad28 Author: James Sirota <jsir...@apache.org> Date: 2017-08-03T21:51:35Z modified: NOTICE --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #662: METRON-1056: Get field types from Elasticsearch
Github user james-sirota commented on a diff in the pull request: https://github.com/apache/metron/pull/662#discussion_r130675495 --- Diff: metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/IndexingDaoIntegrationTest.java --- @@ -27,28 +28,32 @@ import org.json.simple.parser.ParseException; import org.junit.*; +import java.io.IOException; +import java.util.Arrays; +import java.util.Collections; import java.util.List; +import java.util.Map; --- End diff -- @ottobackwards are you ok with this PR? Can this get a +1 from you? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #669: METRON-1064: Make init script OS-agnostic
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/669 +1 by inspection. thanks, ryan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [VOTE] Apache Metron 0.4.0 release
+1 (Binding) * Verified Keys * Verified mvn clean install completed successfully * Verified AWS install of core via Mpack 29.06.2017, 09:14, "Justin Leet" <justinjl...@gmail.com>: > +1 (Non-binding) > > * Verified Keys > * Verified mvn clean install completed successfully > * Ran full dev: saw data flow through, ran a couple of the REST APIs, and > opened up and clicked through a bit of the Management API. > * Examined site-book and didn't see any issues > > On Thu, Jun 29, 2017 at 11:46 AM, Casey Stella <ceste...@gmail.com> wrote: > >> +1 (binding) >> * Verified keys >> * Verified mvn build >> * Verified unit and integration tests run >> * Verified license check runs >> * Verified fulldev spun up with smoketest >> >> On Wed, Jun 28, 2017 at 8:10 PM, Anand Subramanian < >> asubraman...@hortonworks.com> wrote: >> >> > +1 (non-binding) >> > >> > * Brought up Metron stack on 12-node CentOS7 openstack cluster >> > * Verify all services come up fine [PASS] >> > * Bro, YAF and snort - ingest into respective kafka topics and write >> > indices [PASS] >> > * Add squid telemetry, ingest into kafka topic and write indices [PASS] >> > * Metron YAF Zeppelin dashboard with sample ingested YAF data [PASS] >> > * Management UI and REST Swagger UI sanity check [PASS] >> > >> > >> > -Anand >> > >> > >> > >> > >> > >> > On 6/28/17, 12:06 AM, "Matt Foley" <ma...@apache.org> wrote: >> > >> > >This is a call to vote on releasing this rc4 as “Apache Metron 0.4.0”. >> > >(Note: this is rc4 because the release candidate needed to be modified >> > with another commit after the rc3 tag was pushed to public.) >> > > >> > >Full list of changes in this release: >> > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/RELEASE_NOTES >> > > >> > >The tag/commit to be voted upon is: >> > >d52f574f8294e453ecad3871526858a0c3c2033d (tag apache-metron-0.4.0-rc4) >> > > >> > >The source archive being voted upon can be found here: >> > >https://dist.apache.org/repos/dist/dev/metron/0.4.0- >> > RC4/apache-metron-0.4.0-rc4.tar.gz >> > >and in github at: >> > >https://github.com/apache/metron/tree/Metron_0.4.0 >> > > >> > >Other release files, signatures and digests can be found here: >> > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS >> > > >> > >The release artifacts are signed with the following key: >> > >https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS >> > >pub rsa4096/4169AA27ECB31663 2011-07-31 [SCEA] >> > >Key fingerprint = 7854 36A7 8258 6B71 829C 67A0 4169 AA27 ECB3 1663 >> > >uid = Matthew Foley (CODE SIGNING KEY) <ma...@apache.org> >> > > >> > >Please vote on releasing this package as Apache Metron 0.4.0. >> > >When voting, please list the actions taken to verify the release. >> > > >> > >Recommended build validation and verification instructions are posted >> > here: >> > >https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds >> > > >> > >This vote will be open for at least 72 hours. Please vote one of the >> > following responses: >> > >+1 Release this package as Apache Metron 0.4.0-RC4 >> > >0 No opinion >> > >-1 Do not release this package because... >> > > >> > >Thank you, >> > >--Matt >> > >(your friendly release manager) >> > > >> > > >> > > >> > --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
Re: [INCOMING] Metron 0.4.0 release (RC3)
Matt, I think we have been waiting for a reply from Christian for a while. I would like to proceed with the build and if he chimes in we can include his PR in the next release. 26.06.2017, 10:17, "Matt Foley" <ma...@apache.org>: > Hi all, > I am proceeding with building the 0.4.0 release candidate, from current HEAD > of master. > > Christian, still looking forward to getting the improvements for the PaloAlto > parser in, in testable form. If you’re not going to be able to fix the unit > tests presently, please tell the community so maybe someone else will pick up > the task. (BTW, we are all under various time constraints, so we understand > if you cannot commit the time at this point.) You also mentioned being able > to provide some anonymized test data, which would be extremely useful if > someone else does need to do the unit tests. > > Thanks much, > --Matt > > On 6/2/17, 11:36 AM, "Matt Foley" <mfo...@hortonworks.com on behalf of > ma...@apache.org> wrote: > > Hi Christian, > I agree this would be nice to have. I also agree with @kylerichardson ‘s > review comments that the change (with field renames and outputs) is large > enough to require consistent changes in the unit test. Could you please > revive the unit test for PaloAltoParser? > > Thanks, > --Matt > > On 6/2/17, 2:03 AM, "Christian Tramnitz" <tramn...@trasec.de> wrote: > > While not a must-have, METRON-941 / PR-579 should be trivial enough > to include it. > > Thanks, > Christian --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
Re: [DISCUSS] Mutation of Indexed Data
n Ball < >> > si...@simonellistonball.com> wrote: >> > >> > > I'd say that was an excellent set of requirements (very similar to the >> > one >> > > we arrived on with the last discuss thread on this) >> > > >> > > My vote remains a transaction log in hbase given the relatively low >> > volume >> > > (human scale) i would not expect this to need anything fancy like >> > > compaction into hdfs state, but that does make a good argument for a >> long >> > > term dataframe solution for spark, with a short term stop gap using a >> > > joined data frame and shc. >> > > >> > > Simon >> > > >> > > Sent from my iPhone >> > > >> > > > On 22 Jun 2017, at 05:11, Otto Fowler <ottobackwa...@gmail.com> >> wrote: >> > > > >> > > > Can you clarify what data stores are at play here? >> > > > >> > > > >> > > > On June 21, 2017 at 17:07:42, Casey Stella (ceste...@gmail.com) >> wrote: >> > > > >> > > > Hi All, >> > > > >> > > > I know we've had a couple of these already, but we're due for another >> > > > discussion of a sensible approach to mutating indexed data. The >> > > motivation >> > > > for this is users will want to update fields to correct and augment >> > data. >> > > > These corrections are invaluable for things like feedback for ML >> models >> > > or >> > > > just plain providing better context when evaluating alerts, etc. >> > > > >> > > > Rather than posing a solution, I'd like to pose the characteristics >> of >> > a >> > > > solution and we can fight about those first. ;) >> > > > >> > > > In my mind, the following are the characteristics that I'd look for: >> > > > >> > > > - Changes should be considered additional or replacement fields for >> > > > existing fields >> > > > - Changes need to be available in the web view in near real time (on >> > the >> > > > order of milliseconds) >> > > > - Changes should be available in the batch view >> > > > - I'd be ok with eventually consistent with the web view, thoughts? >> > > > - Changes should have lineage preserved >> > > > - Current value is the optimized path >> > > > - Lineage search is the less optimized path >> > > > - If HBase is part of a solution >> > > > - maintain a scan-free solution >> > > > - maintain a coprocessor-free solution >> > > > >> > > > Most of what I've thought of is something along the lines: >> > > > >> > > > - Diffs are stored in columns in a HBase row(s) >> > > > - row: GUID:current would have one column with the current >> > > > representation >> > > > - row: GUID:lineage would have an ordered set of columns representing >> > > > the lineage diffs >> > > > - Mutable indices is directly updated (e.g. solr or ES) >> > > > - We'd probably want to provide transparent read support downstream >> > > > which supports merging for batch read: >> > > > - a spark dataframe >> > > > - a hive serde >> > > > >> > > > What I'd like to get out of this discussion is an architecture >> document >> > > > with a suggested approach and the necessary JIRAs to split this up. >> If >> > > > anyone has suggestions or comments about any of this, please speak >> up. >> > > I'd >> > > > like to actually get this done in the near-term. :) >> > > > >> > > > Best, >> > > > >> > > > Casey >> > > >> > >> -- >> >> Jon --- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
[GitHub] incubator-metron issue #573: METRON-943: Create traffic connections report i...
Github user james-sirota commented on the issue: https://github.com/apache/incubator-metron/pull/573 perfect. + 1 on the dashboard. Looks like travis failed, though --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #556: METRON-903: Create a connections report in Zepp...
Github user james-sirota commented on the issue: https://github.com/apache/incubator-metron/pull/556 Thanks for fixing the cumulative report. The histogram looks great +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #561: METRON-913: Create IP Report in Zeppelin
Github user james-sirota commented on the issue: https://github.com/apache/incubator-metron/pull/561 + 1 looks great --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [DISCUSS] Modify bylaws to allow speculative branches
Hi Guys, I would like to put this up for a vote. We would modify the bylaws with the following: Significant, pervasive features are often developed in a speculative branch of the repository. The PMC may grant commit rights on the branch to its consistent contributors, while the initiative is active. Branch committers are responsible for shepherding their feature into an active release and do not cast binding votes or vetoes in the project. The branch committers are also responsible for filing Jiras related to the development and should associate branch commits to appropriate Jiras. The acceptance criteria and the review process for a branch commit would be relaxed, limited primarily to top-level conceptual review, until the branch is merged back into master. Thanks, James 24.04.2017, 14:55, "James Sirota" <jsir...@apache.org>: > The concrete example where this would be used is the Alerts UI branch that > Raghu is currently working on: > https://github.com/iraghumitra/metron-alerts > > It would require a lot of feedback from the community and would need to > iterate rapidly. Branch review in this sense would mean that people take a > top-level glance at the UI and provide feedback. A more formal review would > happen prior to the branch being merged back into master. > > Thanks, > James > > 24.04.2017, 09:17, "Justin Leet" <justinjl...@gmail.com>: >> I'd prefer to still keep "Review then Commit" but I'd be interested in >> lowering the review bar for it as long as the branch gets a solid final >> review before merge. I don't think a feature branch needs to be >> micromanaged, but people should still take a quick glance at the commits >> that come through. My stance on this may change if we find feature commits >> stagnating for periods of time. >> >> Basically, what I'd be looking for out of a feature branch is something >> that's: >> * Implementing a feature too large or with too much experimentation for a >> single PR. >> * Is at least looked at semi-regularly to have the opportunity for the >> community to point out issues or scalability concerns without minor >> concerns being blockers to progress. Otherwise, it's no different from a >> giant PR that doesn't get examined until the PR is made. >> * Reviews should only very rarely be blockers to progress continuing along >> the branch. The branch shouldn't grind to a halt periodically or even >> significantly slow progress to cleanup syntax or similar things. The >> tradeoff for this is I'd expect refactoring in a lot of cases before it >> hits master to catch up on any of these things that we do care about, but >> that aren't as important while the branch gets built up initially. >> * Kept reasonably up to date with master. I really don't want a feature >> branch sitting out for three months without being synced and then suddenly >> everything explodes when we try to bring it back. It just causes all sorts >> of review and testing issues. >> * Ideally managed so that squashing multiple people's commits appropriately >> is not a horrible nightmare. >> * (Hopefully) able to be merged into master more smoothly as it's gotten >> the major concerns cleaned out along the way. >> >> Ideally, we also use this sort of framework to avoid some of the large PRs >> we've seen around different features that felt very hard to get a grasp of >> and review. >> >> I'm pretty ambivalent on branch committers, and I'd be curious to get some >> opinions (or possibly experiences if anyone has some) before I form a solid >> opinion. >> >> On Mon, Apr 24, 2017 at 12:01 PM, Michael Miklavcic < >> michael.miklav...@gmail.com> wrote: >> >>> "Branch committers ... do not cast binding votes or vetoes in the >>> project." >>> It sounds similar to what we do for other PRs except in this case it's >>> 1..n >>> contributors instead of just one. The step for getting the feature branch >>> merged into trunk sounds reasonable and clear enough, but as Casey >>> mentioned I'm also curious what the commit/review lifecycle looks like >>> while the feature branch is active. >>> >>> On Fri, Apr 21, 2017 at 4:00 PM, Casey Stella <ceste...@gmail.com> wrote: >>> >>> > So, what would that look like from a practical perspective? >>> > * I presume commits would still associate to a JIRA, right? >>> > * Are you proposing changing the strategy from Review then Commit to >>> Commit >>> > then Review for these branches? >&g
