Re: [DISCUSS] Deprecate Least Recently Used Pruner

2019-09-03 Thread James Sirota
nd much documentation on how to use >>  > this feature. I certainly don't know the entire user community, so please >>  > let me know if anyone is using this functionality or believes that it >>  > should be maintained going forward. >>  > >>  > Would you support deprecating this feature? >>  > >>  > Thanks >>  > ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

2019-09-03 Thread James Sirota
ot be performed the same way we've done it in the >>  past. >>  >>> A number of platform upgrades, including OS, are required: >>  >>> >>  >>> 1. Requires the OS to be updated on all nodes because there are no >>  >>> Centos6

Re: [DISCUSS] Upgrade to HDP 3.1.0

2019-09-03 Thread James Sirota
so I reached out to the >>  HBase community to get some guidance. The feedback we received suggests >>  that managing our connections this way should be sufficient. And the HBase >>  connection objects are threadsafe, to boot. >>   >> https://lists.apache.org/

Re: [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images

2019-05-02 Thread James Sirota
  1. https://issues.apache.org/jira/browse/METRON-2099 > > We require a minimum of 72 hours for a vote, not typically including > weekend days. I'd like to leave this vote open until Wednesday 5/8, 12PM > EDT. Please vote +1, -1, or 0 to abstain, and also indicate if your vote is > binding or non-binding. ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

Re: [DISCUSS] Add ngrx to handle state management in Angular

2018-11-26 Thread James Sirota
roach to integration, I don't think we necessarily need a >>  > big refactoring right off the bat. I feel something like this can be done >>  > in a piecemeal approach over time. I think we can start by introducing it >>  > into the project the next time we have a new application feature. >>  > >>  > What are everyone's thoughts around this? >>  > >>  > Cheers, >>  > Shane >>  > >>  > ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

[ANNOUNCE] Shane Ardell is a committer

2018-11-19 Thread James Sirota
there is no need to go via the patch submission process. This should enable better productivity. Being a PMC member enables assistance with the management and to guide the direction of the project. ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

Re: [DISCUSS] Deprecating MySQL

2018-11-15 Thread James Sirota
Which uses a SQL data store. Does > this actually solve the problem of "customers won't install Metron bc SQL > store?" or are there other issues we need to address? > > On Thu, Nov 15, 2018 at 9:30 AM James Sirota wrote: > >>  Hi Guys, >> >>  My op

Re: [DISCUSS] Deprecate split-join enrichment topology in favor of unified enrichment topology

2018-11-15 Thread James Sirota
t;>  > are >>>  > > some >>>  > > > > > simple reasons to deprecate the split-join topology. >>>      > > > > > >>>  > > > > > 1. Unified topology performs better. >>>  > > > > > 2. The configuration, especially for performance >>>  tuning is >>>  > much, >>>  > > > much >>>  > > > > > simpler in the unified model. >>>  > > > > > 3. The footprint within the cluster is smaller. >>>  > > > > > 4. One of the first activities for any install is >>>  that we >>>  > spend >>>  > > time >>>  > > > > > instructing users to switch to the unified topology. >>>  > > > > > 5. One less moving part to maintain. >>>  > > > > > >>>  > > > > > I'd like to recommend that we deprecate the split-join >>>  > topology and >>>  > > > make >>>  > > > > > the unified enrichment topology the new default. >>>  > > > > > >>>  > > > > > Best, >>>  > > > > > Mike >>>  > > > > > >>>  > > > > >>>  > > > >>>  > > >>>  > >>>  > >>>  > -- >>> >>>  Jon Zeolla ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

Re: [DISCUSS] Deprecating MySQL

2018-11-15 Thread James Sirota
 and then strip out the RDBMS stuff in the following. We would continue to >>>  use LDAP for users and HBase for non-LDAPy user settings (as we currently >>>  do). We should also provide a small demo LDAP for full dev. Since we are >>>  looking at adding Knox into the stack, that project provides a convenient >>>  mini-LDAP demo service which would do this job without the need to add >>>  additional components. >>> >>>  Thoughts? Anyone relying on MySQL for users (if so, are you aware that your >>>  passwords are all plaintext? How do you currently handle the shortcomings >>>  and admin overhead?) Any objections? >>> >>>  Simon ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

Re: [DISCUSS] Knox SSO feature branch review and features

2018-11-15 Thread James Sirota
ank you Simon. >>  > >> > > > >>  > >> > > > I need to redact my initial list: >>  > >> > > > >>  > >> > > > 1. Node migrated to Spring Boot, expressjs migrated to a >>  > >> > > > non-JS/non-NodeJs proxying mechanism (ie Zuul in this case) >>  > >

Re: [DISCUSS] Batch Profiler Feature Branch

2018-09-27 Thread James Sirota
ns? >>  > >> > >> > > > > > >>  > >> > >> > > > > > You would just get a profile that is slightly different >>  > >> over >>  > >> > the >>  > >> > &g

Re: [DISCUSS] Knox SSO feature branch review and features

2018-09-19 Thread James Sirota
4. Introduction of Netflix's Zuul. >>>  > > https://issues.apache.org/jira/browse/METRON-1665. >>>  > > - > "The UIs currently proxy to the REST API to avoid CORS >>>  issues, >>>  > > this will be achieved with Zuul." >>>  > > - Can we elaborate more on where or how CORS is a problem with >>>  our >>>  > > existing architecture, how Zuul will help solve that, and how it >>>  > > fits with >>>  > > Knox? Wouldn't this be handled by Knox? Since Larry McCay >>>  chimed in >>>  > > with >>>  > > interest on the original SSO thread about the FB, I'm hoping he >>>  is >>>  > > also >>>  > > willing to chime in on this as well. >>>  > > - This looks like it has the potential to be a rather large >>>  piece >>>  > of >>>  > > fundamental infrastructure (as it's also pertinent to >>>  > microservices) >>>  > > to >>>  > > pull into the platform, and I'd like to be sure the community is >>>  > > aware of >>>  > > and is OK with the implications. >>>  > > 5. > "The proposal is to use a spring boot application, allowing >>>  us to >>>  > > harmonize the security implementation across the UI static servers >>>  and >>>  > > the >>>  > > REST layer, and to provide a routing platform for later >>>  > microservices." >>>  > > - >>>  > > https://issues.apache.org/jira/browse/METRON-1665. >>>  > > - Microservices is a pretty loaded term. I know there had been >>>  some >>>  > > discussion a while back during the PCAP feature branch start, >>>  but I >>>  > > don't >>>  > > recall ever reaching a consensus on it. More detail in this >>>  thread >>>  > - >>>  > > >>>  > > >>>  > >>>   >>> https://lists.apache.org/thread.html/1db7c6fa1b0f364f8c03520db9989b4f7a446de82eb4d9786055048c@%3Cdev.metron.apache.org%3E >>>  > > . >>>  > > Can we get some clarification on what is meant by microservices >>>  > > in the case >>>  > > of this FB and relevant PR's, what that architecture looks like, >>>  > and >>>  > > how >>>  > > it's achieved with the proposed changes in this PR/FB? It seems >>>  > Zuul >>>  > > is >>>  > > also pertinent to this discussion, but there are many ways to >>>  > > skin this cat >>>  > > so I don't want to presume - >>>  > > >>>  > > >>>  https://blog.heroku.com/using_netflix_zuul_to_proxy_your_microservices >>>  > > 6. Zuul, Spring Boot, and microservices - Closely related to >>>  > point 5 >>>  > > above. It seems that we weren't quite ready for this when it was >>>  > > brought up >>>  > > in May, or at the very least we had some concern of what direction >>>  to >>>  > > go. >>>  > > What is the operational impact, mpack impact, and how we propose to >>>  > > manage >>>  > > it with Kerberos, etc.? >>>  > > >>>  > > >>>  > >>>   >>> https://lists.apache.org/thread.html/c19904681e6a6d9ea3131be3d1a65b24447dca31b4aff588b263fd87@%3Cdev.metron.apache.org%3E >>>  > > >>>  > > There is a lot to like in this feature branch, imo. Great feature >>>  > addition >>>  > > with Knox and SSO. Introduction of LDAP support for authentication for >>>  > > Metron UI's. Simplification/unification of our server hosting >>>  > > infrastructure. I'm hoping we can flesh out some of the details >>>  pointed >>>  > out >>>  > > above a bit more and get this feature through. Great work so far! >>>  > > >>>  > > Best, >>>  > > Mike Miklavcic >>>  > > >>>  > >> >>  -- >>  -- >>  simon elliston ball >>  @sireb > > -- > -- > simon elliston ball > @sireb ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

Re: [DISCUSS] Batch Profiler Feature Branch

2018-09-19 Thread James Sirota
he Spark History directory in HDFS. > > export HADOOP_USER_NAME=hdfs > hdfs dfs -mkdir /spark2-history > >   4. Change the default input path to `hdfs://localhost:8020/...` to > match the port defined by HDP, instead of port 9000. > > [1] https://issu

Re: [DISCUSS] Migrate from Protractor to Cypress

2018-09-19 Thread James Sirota
This article comparing the two is not favorable for Cypress. Are any of these concerns relevant to us? If not, then I think Cypress is fine https://hackernoon.com/cypress-io-vs-protractor-e2e-testing-battle-d124ece91dc7

Re: [DISCUSS] PCAP data for testing and development

2018-09-19 Thread James Sirota
I think another reason why we removed it was that it was being flagged by antivirus tools. I am not sure that loop and stop would do anything because the resources would still be taken up by idle topologies and idle sensors. I think when we switch to containers and don't have to eat the

Re: [DISCUSS] Internal Metron fields

2018-09-11 Thread James Sirota
with: source:type >>>  and threat:triage:score in metaalerts. >>> >>>  Is it worth considering converting these to internal Metron fields so that >>>  they stay constant and this isn't a problem in the future? I could see >>>  these fields following the sam

Re: [VOTE] Metron Release Candidate 0.6.0-RC1

2018-09-11 Thread James Sirota
September 12 2018, > to account for the weekend. > > [ ] +1 Release this package as Apache Metron 0.3.0-RC1 > > [ ] 0 No opinion > > [ ] -1 Do not release this package because... ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

Re: [DISCUSS] Pcap query branch completion

2018-08-16 Thread James Sirota
ibe what you think is needed here? Each Metron user could >>  >> have different volumes of pcap data spread out over different time >>  >> periods. Are you saying we should limit the data range to something >>  either >>  >> >>  >> constant or configurable? Are we sure all users would want this? Am I >>  >> misinterpreting this requirement? >>  >> >>  >> - UI should manage a queue/history of jobs >>  >> >>  >> What should we document here? Reading that bullet point again, it's sort >>  >> of vague and not very description. What I am referring to is a design >>  that >>  >> >>  >> provides users a way to view and manage jobs in the UI. Currently jobs >>  can >>  >> >>  >> only be run 1 at a time and progress is shown with a status bar, so it's >>  >> somewhat interactive. >>  >> >>  >> - Documentation/blueprint for YARN configuration >>  >> >>  >> >>  > ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

Re: Pcap Query Panel feature branch status

2018-08-08 Thread James Sirota
elevant to the feature > branch but have not been made subtasks should be converted. > >    - Open the Jira >    - select "More" >    - choose "convert to subtask." >    - Search for METRON-1554 in the search box and select the Pcap epic that >    shows up. >

Re: [DISCUSS] Metron Parsers in Nifi

2018-08-08 Thread James Sirota
files). >>>  - The same controller service can be used by all Processors to manage >>>  configs in a consistent manner. >>> >>>  I think controller services would make sense where needed, I’m just not >>>  sure what you imagine them being needed for? >&g

Re: [DISCUSS] Merging Solr feature branch (METRON-1416) into master

2018-06-27 Thread James Sirota
imarily effects the alerts UI. >>>>  > > > >>>>  > > > As the branch has grown and diverged from master, it's gotten >>>>  > > increasingly >>>>  > > > unwieldy to maintain (and I think it's worth a follow-on discussion >>>>  > about >>>>  > > > how we manage refactorings that happen in these sorts of >>>>  branches). I >>>>  > > know >>>>  > > > there's been at least a couple merges from master that have been >>>>  > > > nontrivially difficult and required careful testing, particularly >>>>  > around >>>>  > > > the DAO layer, to avoid regressions in both code and tests. >>>>  > > > >>>>  > > > The feature set is pretty complete. The UI works, barring the >>>>  > metaalert >>>>  > > > issue. Much of the backend has been refactored and seen improved >>>>  test >>>>  > > > coverage benefiting both Solr and Elasticsearch. The main >>>>  difference >>>>  > > > between ES and Solr is the lack of the equivalent visualizations to >>>>  > > > Kibana. I don't believe the feature branch needs to wait for this, >>>>  as >>>>  > > it's >>>>  > > > pretty standalone work that can be added as usage and demand >>>>  dictates. >>>>  > > > >>>>  > > > I'm of the opinion that the benefits of getting the branch into >>>>  master >>>>  > > > outweighs the issues still present, especially in terms of making >>>>  > > > refactoring and features available and easing the dev burden. The >>>>  > > > remaining tickets are Solr specific, and ES functions as it does in >>>>  > > master. >>>>  > > > >>>>  > > > Are there any must-haves before we bring this branch back? Are >>>>  there >>>>  > any >>>>  > > > other concerns we have before a final PR is opened (pending >>>>  completion >>>>  > of >>>>  > > > active PRs and any other must-haves)? >>>>  > > > >>>>  > > > Justin >>>>  > > > >>>>  > > >>>>  > ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

CVE-2018-1273 fixed in Metron 0.5.0

2018-06-26 Thread James Sirota
The following CVE was fixed in Metron 0.5.0: [CVEID]: CVE-2018-1273 [PRODUCT]:Spring Data Commons [VERSION]: versions prior to 1.13 to 1.13.10, 2.0 to 2.0.5, and older [PROBLEMTYPE]:remote code execution attack [REFERENCES]: https://pivotal.io/security/cve-2018-1273 [DESCRIPTION]: Spring Data

Re: [DISCUSS] Update Metron Elasticsearch index names to metron_

2018-01-26 Thread James Sirota
i am +1 on it then 26.01.2018, 15:56, "Michael Miklavcic" <michael.miklav...@gmail.com>: > Just checked on the length issue - we should be good - > https://github.com/elastic/elasticsearch/issues/8079 > > On Fri, Jan 26, 2018 at 3:37 PM, James Sirota <jsir...

Re: [DISCUSS] Time to remove github updates from dev?

2018-01-26 Thread James Sirota
is particularly >>  helpful >>  > for those reading the list from a list aggregation service. >>  > >>  > Cheers >>  > >>  > >>  > [1] https://lists.apache.org/list.html?iss...@nifi.apache.org >>  > > > -- > > Jon ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

Re: [DISCUSS] Update Metron Elasticsearch index names to metron_

2018-01-26 Thread James Sirota
to a standard prefix for all Metron indices. I've had the same >>>>  thought >>>>>  myself and you laid out the advantages well. >>>>> >>>>>  On Wed, Jan 24, 2018 at 3:47 PM zeo...@gmail.com <zeo...@gmail.com> >>>>  wrote: >&g

Re: Metron User Community Meeting Call

2018-01-26 Thread James Sirota
and knowledge sharing as opposed to technical >>  > >> discussion or implementation details from members of the Apache >>  > Metron >>  > >> Community >>  > >> - >>  > >> >>  > >> Existing Feature demonstrations >

Re: Metron Alert UI and zero-down time Elasticsearch re-index

2018-01-03 Thread James Sirota
part of Alert-UI > because we need to change it to refer to the alias instead of the old index > name. Please advise how it can be covered in the older version of Metron > Alert-UI. > > Regards, > Ali ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

Re: [DISCUSS] Generating and Interacting with serialized summary objects

2018-01-03 Thread James Sirota
t this idea too. >>>  >>>> >>>  >>>> >>>  >>>> On Sun, Dec 24, 2017 at 8:20 PM, Casey Stella <ceste...@gmail.com> >>>  >>> wrote: >>>  >>>> >>>  >>>>> Hi all, >>>  >>>>> >>>  >>>>> I wanted to get some feedback on a sensible plan for something. It >>>  >>>>> occurred to me the other day when considering the use-case of >>>  >> detecting >>>  >>>>> typosquatted domains, that one approach was to generate the set of >>>  >>>>> typosquatted domains for some set of reference domains and compare >>>  >>>> domains >>>  >>>>> as they flow through. >>>  >>>>> >>>  >>>>> One way we could do this would be to generate this data and import >>>  >> the >>>  >>>>> typosquatted domains into HBase. I thought, however, that another >>>  >>>> approach >>>  >>>>> which may trade-off accuracy to remove the network hop and potential >>>  >>> disk >>>  >>>>> seek by constructing a bloom filter that includes the set of >>>  >>> typosquatted >>>  >>>>> domains. >>>  >>>>> >>>  >>>>> The challenge was that we don't have a way to do this currently. We >>>  >>> do, >>>  >>>>> however, have a loading infrastructure (e.g. the flatfile_loader) >>>  and >>>  >>>>> configuration (see https://github.com/apache/ >>>  >>> metron/tree/master/metron- >>>  >>>>> platform/metron-data-management#common-extractor-properties) which >>>  >>>>> handles: >>>  >>>>> >>>  >>>>> - parsing flat files >>>  >>>>> - transforming the rows >>>  >>>>> - filtering the rows >>>  >>>>> >>>  >>>>> To enable the new use-case of generating a summary object (e.g. a >>>  >> bloom >>>  >>>>> filter), in METRON-1378 (https://github.com/apache/metron/pull/879) >>>  >> I >>>  >>>>> propose that we create a new utility that uses the same extractor >>>  >>> config >>>  >>>>> add the ability to: >>>  >>>>> >>>  >>>>> - initialize a state object >>>  >>>>> - update the object for every row >>>  >>>>> - merge the state objects (in the case of multiple threads, in the >>>  >>>>> case of one thread it's not needed). >>>  >>>>> >>>  >>>>> I think this is a sensible decision because: >>>  >>>>> >>>  >>>>> - It's a minimal movement from the flat file loader >>>  >>>>> - Uses the same configs >>>  >>>>> - Abstracts and reuses the existing infrastructure >>>  >>>>> - Having one extractor config means that it should be easier to >>>  >>>>> generate a UI around this to simplify the experience >>>  >>>>> >>>  >>>>> All that being said, our extractor config is..shall we >>>  say...daunting >>>  >>> :). >>>  >>>>> I am sensitive to the fact that this adds to an existing difficult >>>  >>>> config. >>>  >>>>> I propose that this is an initial step forward to support the >>>  >> use-case >>>  >>>> and >>>  >>>>> we can enable something more composable going forward. My concern >>>  in >>>  >>>>> considering this as the first step was that it felt that the >>>  >> composable >>>  >>>>> units for data transformation and manipulation suddenly takes us >>>  >> into a >>>  >>>>> place where Stellar starts to look like Pig or Spark RDD API. I >>>  >> wasn't >>>  >>>>> ready for that without a lot more discussion. >>>  >>>>> >>>  >>>>> To summarize, what I'd like to get from the community is, after >>>  >>> reviewing >>>  >>>>> the entire use-case at https://github.com/cestella/ >>>  >>>> incubator-metron/tree/ >>>  >>>>> typosquat_merge/use-cases/typosquat_detection: >>>  >>>>> >>>  >>>>> - Is this so confusing that it does not belong in Metron even as a >>>  >>>>> first-step? >>>  >>>>> - Is there a way to extend the extractor config in a less >>>  >> confusing >>>  >>>>> way to enable this? >>>  >>>>> >>>  >>>>> I apologize for making the discuss thread *after* the JIRAs, but I >>>  >> felt >>>  >>>>> this one might bear having some working code to consider. >>>  >>>>> >>>  >>>> >>>  >>> >>>  >> ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

Re: Metron - Emailing Alerts

2017-12-13 Thread James Sirota
create the right batching mechanism (at > a cost of possible higher latency than you might get with a more specific > alert batcher?) > > Simon > >>  On 13 Dec 2017, at 21:23, James Sirota <jsir...@apache.org> wrote: >> >>  I agree with Simon. If you email each

Re: [DISCUSS] Community Meetings

2017-12-13 Thread James Sirota
USS] Community Meetings >>>>> >>>>>  I think that we all want to have regular community meetings. We may be >>>>>  better able to keep to a regular schedule with these meetings if we >>>>  spread >>>>>  out the responsibility for them from James and Casey, both of whom > > have >>>  a >>>>>  lot on their plate already. >>>>> >>>>>  I would be willing to coordinate and run the meetings, and would > > welcome >>>>>  anyone else who wants to help when they can. >>>>> >>>>>  The only issue for me is I do not have a web-ex account that I can use >>>  to >>>>>  hold the meeting. So I’ll need some recommendations for a suitable >>>>>  alternative. I have not been able to find an Apache Friendly >>>  alternative, >>>>>  in the same way that Atlassian is apache friendly. >>>>> >>>>>  So - from what I can see we need to: >>>>> >>>>>  - Talk through who is going to do it >>>>>  - How are we going to host it >>>>>  - When are we going to do it >>>>> >>>>>  Anything else? >>>>> >>>>>  ottO ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

Re: Metron - Emailing Alerts

2017-12-13 Thread James Sirota
;> >>  -Ahmed >>  ___ >>  Ahmed Shah (PMP, M. Eng.) >>  Cybersecurity Analyst & Developer >>  GCR - Cybersecurity Operations Center >>  Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php> ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

2017-11-27 Thread James Sirota
gin for Bro' is now maintained in the external >>  repository that we set up a while back. >> >> - Metron Core: git://git.apache.org/metron.git >> - Kafka Plugin for Bro: git://git.apache.org/ >> metron-bro-plugin-kafka.git >> >>  (Q) Do we need to change anything in the release procedure to account for >>  this? ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

Re: [DISCUSS] Are/how are you using the ES data pruner?

2017-11-27 Thread James Sirota
ot look like it has any built-in >>  > > scheduling semantics, so I assume this was a cron job. I think that >>  about >>  > > covers it. Anything I've missed? >>  > > >>  > > I'm adding a quick doc write-up to METRON-939 ( >>  > > https://github.com/apache/metron/pull/840) for using Curator to prune >>  > > indices from Elasticsearch. It is desirable to make sure I've covered >>  > > existing use cases. >>  > > >>  > > Best, >>  > > Mike >>  > > >>  > >>  > >>  > >>  > -- >>  > A.Nazemian >>  > > > -- > A.Nazemian ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...

2017-10-26 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/811 +1 from me as well. Great job @justinleet ---

[GitHub] metron issue #796: METRON-1224: Add time range selection to search control

2017-10-26 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 +1 ---

[GitHub] metron issue #796: METRON-1224: Add time range selection to search control

2017-10-26 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 + 1. Gret job. all pass login to application ✓ should display error message for invalid credentials ✓ should login for valid credentials ✓ should logout

[GitHub] metron issue #796: METRON-1224: Add time range selection to search control

2017-10-26 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 login to application ✓ should display error message for invalid credentials ✓ should login for valid credentials ✓ should logout metron-alerts App

[GitHub] metron issue #796: METRON-1224: Add time range selection to search control

2017-10-24 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 @iraghumitra looks like everything has been addressed. I am +1 on my side, but lets have @merrimanr chime in ---

[GitHub] metron issue #796: METRON-1224: Add time range selection to search control

2017-10-23 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 A few things didn't work for me. First, when I select a time range of (t-x minutes) the start and end time does not fill in per screen shot below. https://user

[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...

2017-10-23 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/811 @nickwallen to avoid scope creep on this PR I created a follow-on PR to figure out how to represent meta alerts in the facet panel. https://issues.apache.org/jira/browse/METRON-1276 I

[GitHub] metron issue #811: METRON-1272: Hide child alerts from searches and grouping...

2017-10-23 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/811 @nickwallen what you are looking at is a desired behavior. If the alerts are a part of the meta alert they do not appear in the facets ---

new committer: Raghu Mitra

2017-10-20 Thread James Sirota
The Project Management Committee (PMC) for Apache Metron has invited Raghu Mitra to become a committer and we are pleased to announce that he has accepted. Being a committer enables easier contribution to the project since there is no need to go via the patch submission process. This should

[GitHub] metron issue #803: Metron-1252: Build ui for grouping alerts into meta alert...

2017-10-19 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/803 You should not have empty meta alerts. That does not make sense ---

[GitHub] metron issue #579: METRON-941 fix PaloAltoParser

2017-10-16 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/579 Hi we would like to get this into the next release. @ctramnitz we'll be happy to help you fix it ---

[GitHub] metron issue #710: Metron-1083: Add filters using faceted search capabilitie...

2017-10-13 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/710 Ok, I opened https://issues.apache.org/jira/browse/METRON-1250 as a follow on jira for this ---

[GitHub] metron issue #796: METRON-1224: Add time range selection to search control

2017-10-12 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 I tested the PR. The only issue I see is that when I paste the timestamp or manually type it into the boxes it overwrites it with the calendar entries. So essentially the only way to get

[GitHub] metron issue #796: METRON-1224: Add time range selection to search control

2017-10-12 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/796 @ottobackwards @iraghumitra i already filed a feature request on that: https://issues.apache.org/jira/browse/METRON-1248 ---

[GitHub] metron pull request #788: METRON-1223: Support for adding comments to alerts

2017-10-11 Thread james-sirota
Github user james-sirota commented on a diff in the pull request: https://github.com/apache/metron/pull/788#discussion_r144174823 --- Diff: metron-interface/metron-alerts/src/app/alerts/alert-details/alert-details.component.ts --- @@ -133,6 +173,40 @@ export class

Re: [DISCUSS] Upgrading Elasticsearch from 2.x to 5.x

2017-10-11 Thread James Sirota
;>  > > > *High Level* >>>  > > > >>>  > > > IndexRequest indexRequest = new IndexRequest("posts", "doc", "1") >>>  > > > .source("user", "kimchy", >>>  > > >

Re: Metron 0.4.2 release date

2017-10-09 Thread James Sirota
more detailed timeline in mind, I would >>  > love to hear more. >>  > >>  > Jon >>  > >>  > On Sun, Oct 8, 2017, 09:05 Ali Nazemian <alinazem...@gmail.com> wrote: >>  > >>  > > Hi all, >>  > > >>  > > I was wondering when Metron 0.4.2 will be released and whether it >>  > includes >>  > > Metron-777 and Elasticsearch 5.x or not? >>  > > >>  > > Cheers, >>  > > Ali >>  > > >>  > -- >>  > >>  > Jon >>  > >> >>  -- >>  A.Nazemian > -- > > Jon ---  Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org

Re: who is having problems installing?

2017-10-09 Thread James Sirota
t > accurate. Missing indexing, errors, etc. I'm sure there are plenty more > examples as well, and I don't think it's reasonable to point people to the > wiki almost at all any longer (the squid walk-through is a good example of > something still very valuable) because doing so is o

Re: SUM aggregator not working?

2017-10-06 Thread James Sirota
ieldToTypeMap": {}, >>>          "config": {} >>>  }, >>>  "threatIntel": { >>>  "fieldMap": { >>>  "stellar": { >>>  "config": [ >>>  "is_alert := exists(is_work) >>>  && >>>  is_work != true && eventName == \"ConsoleLogin\"", >>>  "is_alert := is_alert || >>>  (eventName == \"ConsoleLogin\" && >>>  userIdentity:sessionContext:attributes:mfaAuthenticated >>>  == \"False\")", >>>  "is_alert := is_alert || >>>  (eventName == \"ConsoleLogin\" && additionalEventData:MFAUsed == >>>  \"No\")" >>>  ] >>>  } >>>  }, >>>  "fieldToTypeMap": {}, >>>  "config": {}, >>>  "triageConfig": { >>>  "riskLevelRules": [ >>>  { >>>  "name": "Not WORK", >>>  "comment": "Checks whether the >>>  field is_work is true or false.", >>>  "rule": "is_work == false", >>>  "score": 20, >>>  "reason": "FORMAT('%s is not >>>  an >>>  WORK network!', sourceIPAddress)" >>>  }, >>>  { >>>  "name": "MFA", >>>  "comment": "Checks whether MFA >>>  used or not.", >>>  "rule": >>>  "userIdentity:sessionContext:attributes:mfaAuthenticated == 'False'", >>>  "score": 20, >>>  "reason": null >>>  }, >>>  { >>>  "name": "MFA2", >>>  "comment": "Checks whether MFA >>>  used or not.", >>>  "rule": >>>  "additionalEventData:MFAUsed == 'No'", >>>  "score": 20, >>>  "reason": null >>>  } >>>  ], >>>  "aggregator": "SUM", >>>  "aggregationConfig": {} >>>  } >>>  }, >>>  "configuration": {} >>>  } >>> >>>  Any idea why the score isn't 40? I would expect riskLevelRule 1 & 2 to >>>  be >>>  SUMmed? ---  Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org

Re: who is having problems installing?

2017-10-06 Thread James Sirota
h-level context for what is happening in the cluster and where to look > if you're seeing certain types of issues. > > Jon > > On Fri, Oct 6, 2017 at 1:56 PM James Sirota <jsir...@apache.org> wrote: > >>  Hi Guys, >> >>  How about a meeting at 11 AM PST on this?

Re: Cloudtrail use case

2017-10-06 Thread James Sirota
a full AWS Cloudtrail use case >>  to >>  > > >> the Metron documentation? I would roughly consist of: >>  > > >> - Apache NiFi configuration to retrieve Cloudtrail logs from S3 and >>  > > >> send it to Metron via Kafka. >>  > > >> - Complete Metron sensor configuration (enrichment, alerting, >>  etc...) >>  > > for >>  > > >> this. >>  > > >> >>  > > > >>  > > > Sent too soon :( >>  > > > >>  > > > If anyone would be interested in this documentation, where would add >>  > this >>  > > > in the source? >>  > > > >>  > > >>  > ---  Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org

Re: Need suggestion on how to configure HCP Big Data for Development and Testing

2017-10-06 Thread James Sirota
torage). >> >>  Therefore, how to manage all this resources to properly configured HCP? >> >>  Thanks in advance. ---  Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org

Re: Configuring HCP Big Data for Development

2017-10-06 Thread James Sirota
ent for Hortonworks Cybersecurity > Package within this environment. We have Dell PowerEdge VRTX with 4 nodes > and 4 HDD M630 (shared storage) x 25. > > Therefore, how to manage all this resources to properly configured HCP? > > Hope you guys can help me. Thanks in advance. -----

Re: [DISCUSS] Build broken due to transitive dependencies

2017-10-04 Thread James Sirota
repeatable build. >>>> - We set ourselves up for possible license violation without >>>>  knowing >>>> about it (a transitive dependency changes its license) >>>> >>>>  As we stand, we have a release which doesn't not build after we have &

Re: [DISCUSS] Is there a reason for separate Management & Alerts UIs?

2017-10-04 Thread James Sirota
ave the Management & >>  Alerts UI separate? >> >>  Having another option under "Operations" called "Alerts" in the >>  Management UI seems to make more sense to me... If it's because they are >>  called Management UI and Alerts UI, maybe we sh

Re: who is having problems installing?

2017-10-04 Thread James Sirota
t; *Thank you!* > *Caryll* > > On Wed, Oct 4, 2017 at 7:11 AM, James Sirota <jsir...@apache.org> wrote: > >>  Hi Guys, >> >>  How many people do we have with questions about installing Metron? I can >>  take some time later in the week to schedule a meeting an

Re: who is having problems installing?

2017-10-04 Thread James Sirota
possible issues that I will face and how > to solve them > > *Thank you!* > *Caryll* > > On Wed, Oct 4, 2017 at 9:02 AM, Otto Fowler <ottobackwa...@gmail.com> wrote: > >>  Did you mean to send this to users too? >> >>  On October 3, 2017 at 19:12:10, James Sirota

who is having problems installing?

2017-10-03 Thread James Sirota
Hi Guys, How many people do we have with questions about installing Metron? I can take some time later in the week to schedule a meeting and get everyone unstuck ---  Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org

Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST

2017-10-03 Thread James Sirota
happens when we have 2 parsers/sensors with the same name. > If there's ever a parser/sensor repository, this might be an issue. > > On 2017-09-25 17:38, Otto Fowler wrote: >>  11:30 your time. Sorry I have to pick my kids up from school. 2:30 >>  mine. >> >>  On September

Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST

2017-09-25 Thread James Sirota
Oh sorry, didn't notice that. Otto, when is a good time for you? 25.09.2017, 16:35, "zeo...@gmail.com" <zeo...@gmail.com>: > When is the meeting, given Otto mentioned he can't make 10am? Or did that > change > > Jon > > On Mon, Sep 25, 2017, 19:19 James

Re: [DISCUSS] Splitting up the Indexing Topology

2017-09-25 Thread James Sirota
ssertion being true for all healthy > metron installations, the primary con goes away in my mind. > > Anyway, I'm sure I've missed some pros and cons, so it'd be great to hear > community feedback here. Thoughts? ---  Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org

Re: [DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST

2017-09-25 Thread James Sirota
m unavailable until Thursday of > next week, but not necessarily suggesting this gets moved. > > Jon > > On Thu, Sep 21, 2017, 15:04 Otto Fowler <ottobackwa...@gmail.com> wrote: > >>  I can’t make that time, can we make it later in the day? >> >>  On Sept

[DISCUSS] Community meeting on Tuesday, Sept.23 10AM PST

2017-09-21 Thread James Sirota
-668-4493 ---  Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org

Re: [VOTE] Metron Release Candidate 0.4.1-RC4

2017-09-14 Thread James Sirota
 > > >>  > https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/ >>  site-book/index.html >>  > > > >>  > > >Other release files, signatures and digests can be found here: >>  > > >https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/ >>  > > > >>  > > >The release artifacts are signed with the following key: >>  > > >4169 AA27 ECB3 1663 in >>  > > https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/KEYS >>  > > > >>  > > >Please vote on releasing this package as Apache Metron 0.4.1 >>  > > > >>  > > >When voting, please list the actions taken to verify the >>  release. >>  > > > >>  > > >Recommended build validation and verification instructions >>  are posted >>  > > here: >>  > > >https://cwiki.apache.org/confluence/display/METRON/ >>  Verifying+Builds >>  > > > >>  > > >This vote will be open until 5pm PDT Wednesday 13 Sep, due to >>  the >>  > weekend. >>  > > >Thanks, >>  > > >--Matt >>  > > >(release manager) >>  > > > >>  > > > >>  > > > >>  > > >>  > -- >>  > >>  > Jon >>  > ---  Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org

Re: [ANNOUNCE] Metron community meeting

2017-08-23 Thread James Sirota
Hi Guys, apologies about this. I couldn't record yesterday, but Casey posted a synopsis of the meeting. 22.08.2017, 10:27, "James Sirota" <jsir...@apache.org>: > Yes, I will post a recording > > 21.08.2017, 14:57, "Kyle Richardson" <kylerichards..

Re: [ANNOUNCE] Metron community meeting

2017-08-22 Thread James Sirota
<ottobackwa...@gmail.com> > wrote: > >>  Sounds good >> >>  On August 21, 2017 at 09:43:25, James Sirota (jsir...@apache.org) wrote: >> >>  Hi Jon, >> >>  Sure. Lets move it by a day. The reason it's at this time is to give people >>  in India

Re: [ANNOUNCE] Metron community meeting

2017-08-21 Thread James Sirota
l.com" <zeo...@gmail.com>: > Is it possible to reschedule this to later in the day or another day? That > overlaps with the eclipse on the east cost of the US that some people would > like to enjoy. > > Jon > > On Fri, Aug 18, 2017, 13:48 James Sirota <jsir...@apache.o

[ANNOUNCE] Metron community meeting

2017-08-18 Thread James Sirota
://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC=590161912=1 Anyone is welcome to join. ---  Thank you, James Sirota PPMC- Apache Metron (Incubated and Hatched) jsirota AT apache DOT org

[GitHub] metron issue #683: METRON-1084: Management UI web server license should be A...

2017-08-04 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/683 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] metron pull request #682: modified: NOTICE

2017-08-03 Thread james-sirota
GitHub user james-sirota opened a pull request: https://github.com/apache/metron/pull/682 modified: NOTICE ## Contributor Comments [Please place any comments here. A description of the problem/enhancement, how to reproduce the issue, your testing methodology, etc

[GitHub] metron pull request #662: METRON-1056: Get field types from Elasticsearch

2017-08-01 Thread james-sirota
Github user james-sirota commented on a diff in the pull request: https://github.com/apache/metron/pull/662#discussion_r130675495 --- Diff: metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/IndexingDaoIntegrationTest.java --- @@ -27,28 +28,32

[GitHub] metron issue #669: METRON-1064: Make init script OS-agnostic

2017-07-28 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/metron/pull/669 +1 by inspection. thanks, ryan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

Re: [VOTE] Apache Metron 0.4.0 release

2017-06-29 Thread James Sirota
e vote on releasing this package as Apache Metron 0.4.0. >>  > >When voting, please list the actions taken to verify the release. >>  > > >>  > >Recommended build validation and verification instructions are posted >>  > here: >>  > >https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds >>  > > >>  > >This vote will be open for at least 72 hours. Please vote one of the >>  > following responses: >>  > >+1 Release this package as Apache Metron 0.4.0-RC4 >>  > >0 No opinion >>  > >-1 Do not release this package because... >>  > > >>  > >Thank you, >>  > >--Matt >>  > >(your friendly release manager) >>  > > >>  > > >>  > > >>  > ---  Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org

Re: [INCOMING] Metron 0.4.0 release (RC3)

2017-06-26 Thread James Sirota
uot; <tramn...@trasec.de> wrote: > > While not a must-have, METRON-941 / PR-579 should be trivial enough > to include it. > > Thanks, >    Christian ---  Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org

Re: [DISCUSS] Mutation of Indexed Data

2017-06-26 Thread James Sirota
gt;>  > > > - Changes should be available in the batch view >>  > > > - I'd be ok with eventually consistent with the web view, thoughts? >>  > > > - Changes should have lineage preserved >>  > > > - Current value is the optimized path >>

[GitHub] incubator-metron issue #573: METRON-943: Create traffic connections report i...

2017-05-09 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/incubator-metron/pull/573 perfect. + 1 on the dashboard. Looks like travis failed, though --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] incubator-metron issue #556: METRON-903: Create a connections report in Zepp...

2017-05-03 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/incubator-metron/pull/556 Thanks for fixing the cumulative report. The histogram looks great +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] incubator-metron issue #561: METRON-913: Create IP Report in Zeppelin

2017-05-03 Thread james-sirota
Github user james-sirota commented on the issue: https://github.com/apache/incubator-metron/pull/561 + 1 looks great --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

Re: [DISCUSS] Modify bylaws to allow speculative branches

2017-05-03 Thread James Sirota
. The acceptance criteria and the review process for a branch commit would be relaxed, limited primarily to top-level conceptual review, until the branch is merged back into master. Thanks, James 24.04.2017, 14:55, "James Sirota" <jsir...@apache.org>: > The concrete examp