that it allows you to specify the start date and a
>> number
>> > of
>> > > days for lookback from the start date to purge along with a regex
>> pattern
>> > > to match the index name. It also does not look like it has any built-in
>> > >
n the external
>> repository that we set up a while back.
>>
>> - Metron Core: git://git.apache.org/metron.git
>> - Kafka Plugin for Bro: git://git.apache.org/
>> metron-bro-plugin-kafka.git
>>
>> (Q) Do we need to change anything in the release procedure to account for
>> this?
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
>> Ahmed Shah (PMP, M. Eng.)
>> Cybersecurity Analyst & Developer
>> GCR - Cybersecurity Operations Center
>> Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php>
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
t;>>>> From: Otto Fowler
>>>>> Sent: December 11, 2017 4:41 PM
>>>>> To: dev@metron.apache.org
>>>>> Subject: [DISCUSS] Community Meetings
>>>>>
>>>>> I think that we all want to have regular community meetings
chanism (at
> a cost of possible higher latency than you might get with a more specific
> alert batcher?)
>
> Simon
>
>> On 13 Dec 2017, at 21:23, James Sirota wrote:
>>
>> I agree with Simon. If you email each alert individually you will be
>> overw
;> had.
>>> >>>> I'd love to hear thoughts about this idea too.
>>> >>>>
>>> >>>>
>>> >>>> On Sun, Dec 24, 2017 at 8:20 PM, Casey Stella
>>> >>> wrote:
>>> >>>>
>>> >>>>> Hi all,
>>> >>>>>
>>> >>>>> I wanted to get some feedback on a sensible plan for something. It
>>> >>>>> occurred to me the other day when considering the use-case of
>>> >> detecting
>>> >>>>> typosquatted domains, that one approach was to generate the set of
>>> >>>>> typosquatted domains for some set of reference domains and compare
>>> >>>> domains
>>> >>>>> as they flow through.
>>> >>>>>
>>> >>>>> One way we could do this would be to generate this data and import
>>> >> the
>>> >>>>> typosquatted domains into HBase. I thought, however, that another
>>> >>>> approach
>>> >>>>> which may trade-off accuracy to remove the network hop and potential
>>> >>> disk
>>> >>>>> seek by constructing a bloom filter that includes the set of
>>> >>> typosquatted
>>> >>>>> domains.
>>> >>>>>
>>> >>>>> The challenge was that we don't have a way to do this currently. We
>>> >>> do,
>>> >>>>> however, have a loading infrastructure (e.g. the flatfile_loader)
>>> and
>>> >>>>> configuration (see https://github.com/apache/
>>> >>> metron/tree/master/metron-
>>> >>>>> platform/metron-data-management#common-extractor-properties) which
>>> >>>>> handles:
>>> >>>>>
>>> >>>>> - parsing flat files
>>> >>>>> - transforming the rows
>>> >>>>> - filtering the rows
>>> >>>>>
>>> >>>>> To enable the new use-case of generating a summary object (e.g. a
>>> >> bloom
>>> >>>>> filter), in METRON-1378 (https://github.com/apache/metron/pull/879)
>>> >> I
>>> >>>>> propose that we create a new utility that uses the same extractor
>>> >>> config
>>> >>>>> add the ability to:
>>> >>>>>
>>> >>>>> - initialize a state object
>>> >>>>> - update the object for every row
>>> >>>>> - merge the state objects (in the case of multiple threads, in the
>>> >>>>> case of one thread it's not needed).
>>> >>>>>
>>> >>>>> I think this is a sensible decision because:
>>> >>>>>
>>> >>>>> - It's a minimal movement from the flat file loader
>>> >>>>> - Uses the same configs
>>> >>>>> - Abstracts and reuses the existing infrastructure
>>> >>>>> - Having one extractor config means that it should be easier to
>>> >>>>> generate a UI around this to simplify the experience
>>> >>>>>
>>> >>>>> All that being said, our extractor config is..shall we
>>> say...daunting
>>> >>> :).
>>> >>>>> I am sensitive to the fact that this adds to an existing difficult
>>> >>>> config.
>>> >>>>> I propose that this is an initial step forward to support the
>>> >> use-case
>>> >>>> and
>>> >>>>> we can enable something more composable going forward. My concern
>>> in
>>> >>>>> considering this as the first step was that it felt that the
>>> >> composable
>>> >>>>> units for data transformation and manipulation suddenly takes us
>>> >> into a
>>> >>>>> place where Stellar starts to look like Pig or Spark RDD API. I
>>> >> wasn't
>>> >>>>> ready for that without a lot more discussion.
>>> >>>>>
>>> >>>>> To summarize, what I'd like to get from the community is, after
>>> >>> reviewing
>>> >>>>> the entire use-case at https://github.com/cestella/
>>> >>>> incubator-metron/tree/
>>> >>>>> typosquat_merge/use-cases/typosquat_detection:
>>> >>>>>
>>> >>>>> - Is this so confusing that it does not belong in Metron even as a
>>> >>>>> first-step?
>>> >>>>> - Is there a way to extend the extractor config in a less
>>> >> confusing
>>> >>>>> way to enable this?
>>> >>>>>
>>> >>>>> I apologize for making the discuss thread *after* the JIRAs, but I
>>> >> felt
>>> >>>>> this one might bear having some working code to consider.
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
e need to change it to refer to the alias instead of the old index
> name. Please advise how it can be covered in the older version of Metron
> Alert-UI.
>
> Regards,
> Ali
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
gt; >>
>> > >> Existing Feature demonstrations
>> > >> -
>> > >>
>> > >> Proposed Feature demonstrations
>> > >> -
>> > >>
>> > >> Community feedback
>> > >>
>> > >> These meetings are *not* for :
>> > >>
>> > >> -
>> > >>
>> > >> Support discussions. Those are best left to the mailing lists.
>> > >> -
>> > >>
>> > >> Development discussions. There is another type of meeting for that.
>> > >>
>> > >>
>> > >>
>> > >>
>> > >
>> > > --
>> > >
>> > > Jon
>> > >
>> >
>>
>> --
>> Thanks,
>> Andrew
>>
>> Subscribe to my book: Streaming Data <http://manning.com/psaltis
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__manning.com_psaltis&d=DwMGaQ&c=H50I6Bh8SW87d_bXfZP_8g&r=yeB_CytRmKpr9adMUN0qfcwJfnmWAQuHY9inQHsSRow&m=1J5p3hWBZj3Fc4Xy-CytnTi_kafYqRMsY-Ntvr5HlHw&s=0bpm_zlFmlsG6c8Syr9cEsdZrkKhIuV1mwuJypUBIls&e=>>
>> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_pub_andrew-2Dpsaltis_1_17b_306&d=DwMGaQ&c=H50I6Bh8SW87d_bXfZP_8g&r=yeB_CytRmKpr9adMUN0qfcwJfnmWAQuHY9inQHsSRow&m=1J5p3hWBZj3Fc4Xy-CytnTi_kafYqRMsY-Ntvr5HlHw&s=pRAxEAoEHPf7qW3ly5Ye1Cbo2nvjGlUlGx1UBbcRPhs&e=>>
>> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_intent_user-3Fscreen-5Fname-3Ditmdata&d=DwMGaQ&c=H50I6Bh8SW87d_bXfZP_8g&r=yeB_CytRmKpr9adMUN0qfcwJfnmWAQuHY9inQHsSRow&m=1J5p3hWBZj3Fc4Xy-CytnTi_kafYqRMsY-Ntvr5HlHw&s=8ckuW3QNgrz1rYI6eu3yiH09eLd-Msdwyk7CJ13wWMU&e=>>
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
ought
>>>>> myself and you laid out the advantages well.
>>>>>
>>>>> On Wed, Jan 24, 2018 at 3:47 PM zeo...@gmail.com
>>>> wrote:
>>>>>> I agree with having a metron_ prefix for ES indexes, and the timing.
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> On Wed, Jan 24, 2018 at 3:20 PM Michael Miklavcic <
>>>>>> michael.miklav...@gmail.com> wrote:
>>>>>>
>>>>>>> With the completion of https://github.com/apache/metron/pull/840
>>>>>>> (METRON-939: Upgrade ElasticSearch and Kibana), we have the makings
>>>> for
>>>>> a
>>>>>>> major release rev of Metron in the upcoming release (currently
>>>> slotted
>>>>> to
>>>>>>> 0.4.3, I believe). Since there are non-backwards compatible changes
>>>>>>> pertaining to ES indexing, it seems like a good opportunity to
>>>> revisit
>>>>>> our
>>>>>>> index naming standards.
>>>>>>>
>>>>>>> I propose we add a simple prefix "metron_" to all Metron indexes.
>>>> There
>>>>>> are
>>>>>>> numerous reasons for doing so
>>>>>>>
>>>>>>> - removes the likelihood of index name collisions when we perform
>>>>>>> operations on index wildcard names, e.g. "enrichment_*, indexing_*,
>>>>>>> etc.".
>>>>>>> - ie, this allows us to be more friendly in a multi-tenant ES
>>>>>>> environment for relatively low engineering cost.
>>>>>>> - simplifies the Kibana dashboard a bit. We currently needed to
>>>>>> create a
>>>>>>> special index pattern in order to accommodate multi-index pattern
>>>>>>> matching
>>>>>>> across all metron-specific indexes. Using metron_* would be much
>>>>>> simpler
>>>>>>> and less prone to error.
>>>>>>> - easier for customers to debug and identify Metron-specific indexes
>>>>>> and
>>>>>>> associated data
>>>>>>>
>>>>>>> The reason for making these changes now is that we already have
>>>>> breaking
>>>>>>> changes with ES. Leveraging existing indexed data rather than
>>>> deleting
>>>>>>> indexes and starting from scractch already requires a
>>>>>> re-indexing/migration
>>>>>>> step, so there is no additional effort on the part of users if they
>>>>>> choose
>>>>>>> to attempt a migration. It further makes sense with our current work
>>>>>>> towards upgrading Solr.
>>>>>>>
>>>>>>> We already have a battery of integration and manual tests after the
>>>> ES
>>>>>>> upgrade work that can be leveraged to validate the changes.
>>>>>>>
>>>>>>> Mike Miklavcic
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Jon
>>>>
>>>> --
>>>> A.Nazemian
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
the list from a list aggregation service.
>> >
>> > Cheers
>> >
>> >
>> > [1] https://lists.apache.org/list.html?iss...@nifi.apache.org
>> >
>
> --
>
> Jon
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
i am +1 on it then
26.01.2018, 15:56, "Michael Miklavcic" :
> Just checked on the length issue - we should be good -
> https://github.com/elastic/elasticsearch/issues/8079
>
> On Fri, Jan 26, 2018 at 3:37 PM, James Sirota wrote:
>
>> Seems reasonable to me. The
l topologies means that all of
>> > the topologies read/write to Kafka, which produce a bigger load on the
>> > kafka cluster and then a need for way more infrastructure/servers. The
>> cost
>> > is especially true when we speak about TBs of data ingested every day.
>> >
>> > Im sure there were a very good reason, I was just curious.
>> >
>> > Thanks,
>> > Michel
>> >
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
The following CVE was fixed in Metron 0.5.0:
[CVEID]: CVE-2018-1273
[PRODUCT]:Spring Data Commons
[VERSION]: versions prior to 1.13 to 1.13.10, 2.0 to 2.0.5, and older
[PROBLEMTYPE]:remote code execution attack
[REFERENCES]: https://pivotal.io/security/cve-2018-1273
[DESCRIPTION]:
Spring Data C
gt; > > metaalert
>>>> > > > specific issue that primarily effects the alerts UI.
>>>> > > >
>>>> > > > As the branch has grown and diverged from master, it's gotten
>>>> > > increasingly
>>>> > > > unwieldy to maintain (and I think it's worth a follow-on discussion
>>>> > about
>>>> > > > how we manage refactorings that happen in these sorts of
>>>> branches). I
>>>> > > know
>>>> > > > there's been at least a couple merges from master that have been
>>>> > > > nontrivially difficult and required careful testing, particularly
>>>> > around
>>>> > > > the DAO layer, to avoid regressions in both code and tests.
>>>> > > >
>>>> > > > The feature set is pretty complete. The UI works, barring the
>>>> > metaalert
>>>> > > > issue. Much of the backend has been refactored and seen improved
>>>> test
>>>> > > > coverage benefiting both Solr and Elasticsearch. The main
>>>> difference
>>>> > > > between ES and Solr is the lack of the equivalent visualizations to
>>>> > > > Kibana. I don't believe the feature branch needs to wait for this,
>>>> as
>>>> > > it's
>>>> > > > pretty standalone work that can be added as usage and demand
>>>> dictates.
>>>> > > >
>>>> > > > I'm of the opinion that the benefits of getting the branch into
>>>> master
>>>> > > > outweighs the issues still present, especially in terms of making
>>>> > > > refactoring and features available and easing the dev burden. The
>>>> > > > remaining tickets are Solr specific, and ES functions as it does in
>>>> > > master.
>>>> > > >
>>>> > > > Are there any must-haves before we bring this branch back? Are
>>>> there
>>>> > any
>>>> > > > other concerns we have before a final PR is opened (pending
>>>> completion
>>>> > of
>>>> > > > active PRs and any other must-haves)?
>>>> > > >
>>>> > > > Justin
>>>> > > >
>>>> > >
>>>> >
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
stablished NiFi pattern for being able to provide
>>> access to other services needed by a Processor (e.g. databases or large
>>> configurations files).
>>> - The same controller service can be used by all Processors to manage
>>> configs in a consistent manner.
are relevant to the feature
> branch but have not been made subtasks should be converted.
>
> - Open the Jira
> - select "More"
> - choose "convert to subtask."
> - Search for METRON-1554 in the search box and select the Pcap epic that
> sho
>> >> Can you describe what you think is needed here? Each Metron user could
>> >> have different volumes of pcap data spread out over different time
>> >> periods. Are you saying we should limit the data range to something
>> either
>> >>
>> >> constant or configurable? Are we sure all users would want this? Am I
>> >> misinterpreting this requirement?
>> >>
>> >> - UI should manage a queue/history of jobs
>> >>
>> >> What should we document here? Reading that bullet point again, it's sort
>> >> of vague and not very description. What I am referring to is a design
>> that
>> >>
>> >> provides users a way to view and manage jobs in the UI. Currently jobs
>> can
>> >>
>> >> only be run 1 at a time and progress is shown with a status bar, so it's
>> >> somewhat interactive.
>> >>
>> >> - Documentation/blueprint for YARN configuration
>> >>
>> >>
>> >
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
September 12 2018,
> to account for the weekend.
>
> [ ] +1 Release this package as Apache Metron 0.3.0-RC1
>
> [ ] 0 No opinion
>
> [ ] -1 Do not release this package because...
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
I had to deal with: source:type
>>> and threat:triage:score in metaalerts.
>>>
>>> Is it worth considering converting these to internal Metron fields so that
>>> they stay constant and this isn't a problem in the future? I could see
>>> these fiel
I think another reason why we removed it was that it was being flagged by
antivirus tools. I am not sure that loop and stop would do anything because
the resources would still be taken up by idle topologies and idle sensors. I
think when we switch to containers and don't have to eat the overhe
This article comparing the two is not favorable for Cypress. Are any of these
concerns relevant to us? If not, then I think Cypress is fine
https://hackernoon.com/cypress-io-vs-protractor-e2e-testing-battle-d124ece91dc7
E/conf/
>
> 3. Create the Spark History directory in HDFS.
>
> export HADOOP_USER_NAME=hdfs
> hdfs dfs -mkdir /spark2-history
>
> 4. Change the default input path to `hdfs://localhost:8020/...` to
> match the port defined by HDP, instead of po
gt;> > > It seems comparable to a component upgrade, such as moving from ES
>>> 2.x
>>> > > to
>>> > > 5.6.x and upgrading Angular 6.
>>> > > 4. Introduction of Netflix's Zuul.
>>> > > https://issues.apache.org/jira/browse/METRON-1665.
>>> > > - > "The UIs currently proxy to the REST API to avoid CORS
>>> issues,
>>> > > this will be achieved with Zuul."
>>> > > - Can we elaborate more on where or how CORS is a problem with
>>> our
>>> > > existing architecture, how Zuul will help solve that, and how it
>>> > > fits with
>>> > > Knox? Wouldn't this be handled by Knox? Since Larry McCay
>>> chimed in
>>> > > with
>>> > > interest on the original SSO thread about the FB, I'm hoping he
>>> is
>>> > > also
>>> > > willing to chime in on this as well.
>>> > > - This looks like it has the potential to be a rather large
>>> piece
>>> > of
>>> > > fundamental infrastructure (as it's also pertinent to
>>> > microservices)
>>> > > to
>>> > > pull into the platform, and I'd like to be sure the community is
>>> > > aware of
>>> > > and is OK with the implications.
>>> > > 5. > "The proposal is to use a spring boot application, allowing
>>> us to
>>> > > harmonize the security implementation across the UI static servers
>>> and
>>> > > the
>>> > > REST layer, and to provide a routing platform for later
>>> > microservices."
>>> > > -
>>> > > https://issues.apache.org/jira/browse/METRON-1665.
>>> > > - Microservices is a pretty loaded term. I know there had been
>>> some
>>> > > discussion a while back during the PCAP feature branch start,
>>> but I
>>> > > don't
>>> > > recall ever reaching a consensus on it. More detail in this
>>> thread
>>> > -
>>> > >
>>> > >
>>> >
>>>
>>> https://lists.apache.org/thread.html/1db7c6fa1b0f364f8c03520db9989b4f7a446de82eb4d9786055048c@%3Cdev.metron.apache.org%3E
>>> > > .
>>> > > Can we get some clarification on what is meant by microservices
>>> > > in the case
>>> > > of this FB and relevant PR's, what that architecture looks like,
>>> > and
>>> > > how
>>> > > it's achieved with the proposed changes in this PR/FB? It seems
>>> > Zuul
>>> > > is
>>> > > also pertinent to this discussion, but there are many ways to
>>> > > skin this cat
>>> > > so I don't want to presume -
>>> > >
>>> > >
>>> https://blog.heroku.com/using_netflix_zuul_to_proxy_your_microservices
>>> > > 6. Zuul, Spring Boot, and microservices - Closely related to
>>> > point 5
>>> > > above. It seems that we weren't quite ready for this when it was
>>> > > brought up
>>> > > in May, or at the very least we had some concern of what direction
>>> to
>>> > > go.
>>> > > What is the operational impact, mpack impact, and how we propose to
>>> > > manage
>>> > > it with Kerberos, etc.?
>>> > >
>>> > >
>>> >
>>>
>>> https://lists.apache.org/thread.html/c19904681e6a6d9ea3131be3d1a65b24447dca31b4aff588b263fd87@%3Cdev.metron.apache.org%3E
>>> > >
>>> > > There is a lot to like in this feature branch, imo. Great feature
>>> > addition
>>> > > with Knox and SSO. Introduction of LDAP support for authentication for
>>> > > Metron UI's. Simplification/unification of our server hosting
>>> > > infrastructure. I'm hoping we can flesh out some of the details
>>> pointed
>>> > out
>>> > > above a bit more and get this feature through. Great work so far!
>>> > >
>>> > > Best,
>>> > > Mike Miklavcic
>>> > >
>>> >
>>
>> --
>> --
>> simon elliston ball
>> @sireb
>
> --
> --
> simon elliston ball
> @sireb
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
gt;> > > > > > the user runs the batch seeding job, what happens?
>> > >> > >> > > > > >
>> > >> > >> > > > > > You would just get a profile that is slightly different
>> >
> a lot of dependencies to store users, which is a problem much better
>>>> solved
>>>> by LDAP.
>>>>
>>>> Now that we have the option to use LDAP for user storage, I would suggest
>>>> that we deprecate and ultimately remove all the RDBMS and ORM
>>>> dependencies,
>>>> which significantly reduces our dependencies and simplifies deployment and
>>>> long term management of Metron clusters.
>>>>
>>>> So I propose that we deprecate the RDBMS use in the next Apache release,
>>>> and then strip out the RDBMS stuff in the following. We would continue to
>>>> use LDAP for users and HBase for non-LDAPy user settings (as we currently
>>>> do). We should also provide a small demo LDAP for full dev. Since we are
>>>> looking at adding Knox into the stack, that project provides a convenient
>>>> mini-LDAP demo service which would do this job without the need to add
>>>> additional components.
>>>>
>>>> Thoughts? Anyone relying on MySQL for users (if so, are you aware that
>>>> your
>>>> passwords are all plaintext? How do you currently handle the shortcomings
>>>> and admin overhead?) Any objections?
>>>>
>>>> Simon
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
> I've spent some more time reading through Simon's response and
>> the
>> > >> > added
>> > >> > > > sequence diagram. This is definitely helpful - thank you Simon.
>> > >> > > >
>> > >> > > > I need to redact my initial list:
>> > >> > > >
deprecate the RDBMS use in the next Apache release,
>>> and then strip out the RDBMS stuff in the following. We would continue to
>>> use LDAP for users and HBase for non-LDAPy user settings (as we currently
>>> do). We should also provide a small demo LDAP for full dev. Since we are
>>> looking at adding Knox into the stack, that project provides a convenient
>>> mini-LDAP demo service which would do this job without the need to add
>>> additional components.
>>>
>>> Thoughts? Anyone relying on MySQL for users (if so, are you aware that your
>>> passwords are all plaintext? How do you currently handle the shortcomings
>>> and admin overhead?) Any objections?
>>>
>>> Simon
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
ne. Here
>>> > are
>>> > > some
>>> > > > > > simple reasons to deprecate the split-join topology.
>>> > > > > >
>>> > > > > > 1. Unified topology performs better.
>>> > > > > > 2. The configuration, especially for performance
>>> tuning is
>>> > much,
>>> > > > much
>>> > > > > > simpler in the unified model.
>>> > > > > > 3. The footprint within the cluster is smaller.
>>> > > > > > 4. One of the first activities for any install is
>>> that we
>>> > spend
>>> > > time
>>> > > > > > instructing users to switch to the unified topology.
>>> > > > > > 5. One less moving part to maintain.
>>> > > > > >
>>> > > > > > I'd like to recommend that we deprecate the split-join
>>> > topology and
>>> > > > make
>>> > > > > > the unified enrichment topology the new default.
>>> > > > > >
>>> > > > > > Best,
>>> > > > > > Mike
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> >
>>> > --
>>>
>>> Jon Zeolla
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
se dependency? Which uses a SQL data store. Does
> this actually solve the problem of "customers won't install Metron bc SQL
> store?" or are there other issues we need to address?
>
> On Thu, Nov 15, 2018 at 9:30 AM James Sirota wrote:
>
>> Hi Guys,
>>
need to go via the patch submission process. This should enable better
productivity. Being a PMC member enables assistance with the management and to
guide the direction of the project.
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
gt;> > As far as an approach to integration, I don't think we necessarily need a
>> > big refactoring right off the bat. I feel something like this can be done
>> > in a piecemeal approach over time. I think we can start by introducing it
>> > into the project the next time we have a new application feature.
>> >
>> > What are everyone's thoughts around this?
>> >
>> > Cheers,
>> > Shane
>> >
>> >
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
g diagrams
> 1. https://issues.apache.org/jira/browse/METRON-2099
>
> We require a minimum of 72 hours for a vote, not typically including
> weekend days. I'd like to leave this vote open until Wednesday 5/8, 12PM
> EDT. Please vote +1, -1, or 0 to abstain, and also indicate if your vote is
> binding or non-binding.
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
to
>> using a more robust connection management approach, so I reached out to the
>> HBase community to get some guidance. The feedback we received suggests
>> that managing our connections this way should be sufficient. And the HBase
>> connection objects are t
ot be performed the same way we've done it in the
>> past.
>> >>> A number of platform upgrades, including OS, are required:
>> >>>
>> >>> 1. Requires the OS to be updated on all nodes because there are no
>> >>> Ce
s. I also did not find much documentation on how to use
>> > this feature. I certainly don't know the entire user community, so please
>> > let me know if anyone is using this functionality or believes that it
>> > should be maintained going forward.
>> >
>> > Would you support deprecating this feature?
>> >
>> > Thanks
>> >
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/518
+1 I tested it on AWS and was able to get it to work. Great job!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
to be committed into
master.
What do you think?
---
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
I presume commits would still associate to a JIRA, right?
>> > * Are you proposing changing the strategy from Review then Commit to
>> Commit
>> > then Review for these branches?
>> >
>> > I know that we have some people who are active in the Hadoo
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/556
This dashboard is designed to look at flows and we absolutely do want to
count every flow. So UnionALL is the correct implementation here
---
If your project is set up for it, you
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/556
Just spun up the dashboard. The YAF portion is correct. The YAF sensor
produces flow information from A to B and you produce a table that counts and
orders them. Bro is almost
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/556
@justinleet the top-level cumulative dashboard would not be useful because
you are combining flows with DPI with Alerts data. With respect to the
histogram I would do it as a part of
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/559
My only feedback so far is to display the PCAP results as an HTML list
rather than Zeppelin list. Otherwise great job
---
If your project is set up for it, you can reply to this
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/556
Excellent. I see the HTML table and it works great. This is very well
done. +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/559
Excellent. I see the HTML table and it works great. This is very well
done. +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on
. The acceptance criteria and the review process for a branch
commit would be relaxed, limited primarily to top-level conceptual review,
until the branch is merged back into master.
Thanks,
James
24.04.2017, 14:55, "James Sirota" :
> The concrete example where this would be used
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/561
+ 1 looks great
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/556
Thanks for fixing the cumulative report. The histogram looks great +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user james-sirota commented on the issue:
https://github.com/apache/incubator-metron/pull/573
perfect. + 1 on the dashboard. Looks like travis failed, though
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
seconds)
>> > > > - Changes should be available in the batch view
>> > > > - I'd be ok with eventually consistent with the web view, thoughts?
>> > > > - Changes should have lineage preserved
>> > > > - Current value is the optimized path
>> > > > - Lineage search is the less optimized path
>> > > > - If HBase is part of a solution
>> > > > - maintain a scan-free solution
>> > > > - maintain a coprocessor-free solution
>> > > >
>> > > > Most of what I've thought of is something along the lines:
>> > > >
>> > > > - Diffs are stored in columns in a HBase row(s)
>> > > > - row: GUID:current would have one column with the current
>> > > > representation
>> > > > - row: GUID:lineage would have an ordered set of columns representing
>> > > > the lineage diffs
>> > > > - Mutable indices is directly updated (e.g. solr or ES)
>> > > > - We'd probably want to provide transparent read support downstream
>> > > > which supports merging for batch read:
>> > > > - a spark dataframe
>> > > > - a hive serde
>> > > >
>> > > > What I'd like to get out of this discussion is an architecture
>> document
>> > > > with a suggested approach and the necessary JIRAs to split this up.
>> If
>> > > > anyone has suggestions or comments about any of this, please speak
>> up.
>> > > I'd
>> > > > like to actually get this done in the near-term. :)
>> > > >
>> > > > Best,
>> > > >
>> > > > Casey
>> > >
>> >
>> --
>>
>> Jon
---
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
> enough to require consistent changes in the unit test. Could you please
> revive the unit test for PaloAltoParser?
>
> Thanks,
> --Matt
>
> On 6/2/17, 2:03 AM, "Christian Tramnitz" wrote:
>
> While not a must-have, METRON-941 / PR-579 shoul
;> > >Key fingerprint = 7854 36A7 8258 6B71 829C 67A0 4169 AA27 ECB3 1663
>> > >uid = Matthew Foley (CODE SIGNING KEY)
>> > >
>> > >Please vote on releasing this package as Apache Metron 0.4.0.
>> > >When voting, please list the actions take
ersion.
>> c) These version number changes are in master branch. Creation of new
>> branches does not occur until the idea of creating a maintenance branch or
>> a new release branch has been consented by the community.
>>
>> Please share your thoughts and/or vote.
>> Thanks,
>> --Matt
---
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/669
+1 by inspection. thanks, ryan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user james-sirota commented on a diff in the pull request:
https://github.com/apache/metron/pull/662#discussion_r130675495
--- Diff:
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/IndexingDaoIntegrationTest.java
---
@@ -27,28 +28,32
GitHub user james-sirota opened a pull request:
https://github.com/apache/metron/pull/682
modified: NOTICE
## Contributor Comments
[Please place any comments here. A description of the problem/enhancement,
how to reproduce the issue, your testing methodology, etc
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/682
go to /Users/jsirota/Documents/hcp_release/metron-interface/metron-config
and delete the node_modules folder. then run npm install -prod to only pull in
production libraries. then run npm i
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/683
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/643
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
://hortonworks.webex.com/hortonworks/globalcallin.php?serviceType=MC&ED=590161912&tollFree=1
Anyone is welcome to join.
---
Thank you,
James Sirota
PPMC- Apache Metron (Incubated and Hatched)
jsirota AT apache DOT org
, 11:02, "zeo...@gmail.com" :
> Is it possible to reschedule this to later in the day or another day? That
> overlaps with the eclipse on the east cost of the US that some people would
> like to enjoy.
>
> Jon
>
> On Fri, Aug 18, 2017, 13:48 James Sirota wrote:
>
od
>>
>> On August 21, 2017 at 09:43:25, James Sirota (jsir...@apache.org) wrote:
>>
>> Hi Jon,
>>
>> Sure. Lets move it by a day. The reason it's at this time is to give people
>> in India and Europe a chance to attend live.
>>
>> So
Hi Guys, apologies about this. I couldn't record yesterday, but Casey posted a
synopsis of the meeting.
22.08.2017, 10:27, "James Sirota" :
> Yes, I will post a recording
>
> 21.08.2017, 14:57, "Kyle Richardson" :
>> Unfortunately I have a conflict. Wi
> >The site-book is at:
>> > > >
>> > >
>> > https://dist.apache.org/repos/dist/dev/metron/0.4.1-RC4/
>> site-book/index.html
>> > > >
>> > > >Other release files, signatures and dig
er (US/Canada) 1-877-668-4493
---
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Sorry made a mistake. Tuesday is 26th, not 23rd.
Thanks,
James
21.09.2017, 08:06, "James Sirota" :
> Hi Guys,
>
> I'd like to propose a community meeting for this Tuesday, Sept.23 and focus
> on the strategy for getting Metron-777 and related PRs on Otto's feat
of
> next week, but not necessarily suggesting this gets moved.
>
> Jon
>
> On Thu, Sep 21, 2017, 15:04 Otto Fowler wrote:
>
>> I can’t make that time, can we make it later in the day?
>>
>> On September 21, 2017 at 11:40:37, James Sirota (jsir.
assertion being true for all healthy
> metron installations, the primary con goes away in my mind.
>
> Anyway, I'm sure I've missed some pros and cons, so it'd be great to hear
> community feedback here. Thoughts?
---
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Oh sorry, didn't notice that. Otto, when is a good time for you?
25.09.2017, 16:35, "zeo...@gmail.com" :
> When is the meeting, given Otto mentioned he can't make 10am? Or did that
> change
>
> Jon
>
> On Mon, Sep 25, 2017, 19:19 James Sirota wrote:
have 2 parsers/sensors with the same name.
> If there's ever a parser/sensor repository, this might be an issue.
>
> On 2017-09-25 17:38, Otto Fowler wrote:
>> 11:30 your time. Sorry I have to pick my kids up from school. 2:30
>> mine.
>>
>> On Septem
Hi Guys,
How many people do we have with questions about installing Metron? I can take
some time later in the week to schedule a meeting and get everyone unstuck
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
to solve them
>
> *Thank you!*
> *Caryll*
>
> On Wed, Oct 4, 2017 at 9:02 AM, Otto Fowler wrote:
>
>> Did you mean to send this to users too?
>>
>> On October 3, 2017 at 19:12:10, James Sirota (jsir...@apache.org) wrote:
>>
>> Hi Guys,
>>
ct 4, 2017 at 7:11 AM, James Sirota wrote:
>
>> Hi Guys,
>>
>> How many people do we have with questions about installing Metron? I can
>> take some time later in the week to schedule a meeting and get everyone
>> unstuck
>>
>> ---
separate?
>>
>> Having another option under "Operations" called "Alerts" in the
>> Management UI seems to make more sense to me... If it's because they are
>> called Management UI and Alerts UI, maybe we should make it more general
>> and name it M
>>> - We set ourselves up for possible license violation without
>>>> knowing
>>>> about it (a transitive dependency changes its license)
>>>>
>>>> As we stand, we have a release which doesn't not build after we have
>>>> released it and tested it. It seems to me that we should at a
>>>> minimum as a
>>>> stopgap:
>>>>
>>>> - fix the versions of our dependencies so that they are in a
>>>> working
>>>> state
>>>> - consider a point release to get a working build.
>>>>
>>>> I guess my questions to those of us with more javascript and UI
>>>> experience
>>>> is as follows:
>>>>
>>>> - Does fixing the version of our dependencies actually fix the
>>>> problem
>>>> transitively?
>>>> - IF not, then how do we get a version of a build which is
>>>> consistent
>>>> and repeatable and does not expose us to downstream licensing
>>>> issues?
>>>>
>>>> Thanks,
>>>>
>>>> Casey
---
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
ing to install on multiple VMs
> - What OS are you using
> == Ubuntu Xenial and Zesty, but trying to use CentOS
> - How many sensors are you going to be consuming
> == Unidentified. I was doing a test install at the moment.
>
> *Thank you!*
> *Caryll*
>
> On Thu,
; Package within this environment. We have Dell PowerEdge VRTX with 4 nodes
> and 4 HDD M630 (shared storage) x 25.
>
> Therefore, how to manage all this resources to properly configured HCP?
>
> Hope you guys can help me. Thanks in advance.
-------
Thank you,
J
).
>>
>> Therefore, how to manage all this resources to properly configured HCP?
>>
>> Thanks in advance.
---
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
On 2017-10-05 15:45, Laurens Vets wrote:
>> > > >
>> > > >> Hi,
>> > > >>
>> > > >> Would anyone be interested in adding a full AWS Cloudtrail use case
>> to
>> > > >> the Metron documentation? I would
hat is happening in the cluster and where to look
> if you're seeing certain types of issues.
>
> Jon
>
> On Fri, Oct 6, 2017 at 1:56 PM James Sirota wrote:
>
>> Hi Guys,
>>
>> How about a meeting at 11 AM PST on this? Can everyone who needs to make
&g
}
>>> },
>>> "fieldToTypeMap": {},
>>> "config": {}
>>> },
>>> "threatIntel": {
>>> "fieldMap": {
>>> "stellar": {
>>> "config": [
>>> "is_alert := exists(is_work)
>>> &&
>>> is_work != true && eventName == \"ConsoleLogin\"",
>>> "is_alert := is_alert ||
>>> (eventName == \"ConsoleLogin\" &&
>>> userIdentity:sessionContext:attributes:mfaAuthenticated
>>> == \"False\")",
>>> "is_alert := is_alert ||
>>> (eventName == \"ConsoleLogin\" && additionalEventData:MFAUsed ==
>>> \"No\")"
>>> ]
>>> }
>>> },
>>> "fieldToTypeMap": {},
>>> "config": {},
>>> "triageConfig": {
>>> "riskLevelRules": [
>>> {
>>> "name": "Not WORK",
>>> "comment": "Checks whether the
>>> field is_work is true or false.",
>>> "rule": "is_work == false",
>>> "score": 20,
>>> "reason": "FORMAT('%s is not
>>> an
>>> WORK network!', sourceIPAddress)"
>>> },
>>> {
>>> "name": "MFA",
>>> "comment": "Checks whether MFA
>>> used or not.",
>>> "rule":
>>> "userIdentity:sessionContext:attributes:mfaAuthenticated == 'False'",
>>> "score": 20,
>>> "reason": null
>>> },
>>> {
>>> "name": "MFA2",
>>> "comment": "Checks whether MFA
>>> used or not.",
>>> "rule":
>>> "additionalEventData:MFAUsed == 'No'",
>>> "score": 20,
>>> "reason": null
>>> }
>>> ],
>>> "aggregator": "SUM",
>>> "aggregationConfig": {}
>>> }
>>> },
>>> "configuration": {}
>>> }
>>>
>>> Any idea why the score isn't 40? I would expect riskLevelRule 1 & 2 to
>>> be
>>> SUMmed?
---
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
e. Missing indexing, errors, etc. I'm sure there are plenty more
> examples as well, and I don't think it's reasonable to point people to the
> wiki almost at all any longer (the squid walk-through is a good example of
> something still very valuable) because doing so is o
ear more.
>> >
>> > Jon
>> >
>> > On Sun, Oct 8, 2017, 09:05 Ali Nazemian wrote:
>> >
>> > > Hi all,
>> > >
>> > > I was wondering when Metron 0.4.2 will be released and whether it
>> > includes
>> > > Metron-777 and Elasticsearch 5.x or not?
>> > >
>> > > Cheers,
>> > > Ali
>> > >
>> > --
>> >
>> > Jon
>> >
>>
>> --
>> A.Nazemian
> --
>
> Jon
---
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
gt; > *High Level*
>>> > > >
>>> > > > IndexRequest indexRequest = new IndexRequest("posts", "doc", "1")
>>> > > > .source("user", "kimchy",
>>> > > > "postDate", new Date(),
>>> > > > "message", "trying out Elasticsearch");
>>> > > >
>>> > > > *Note*: there are a few ways to do this with the high level API, but
>>> > this
>>> > > > was the most concise for me to offer a comparison of benefits over
>>> the
>>> > > low
>>> > > > level API.
>>> > > >
>>> > > > *Thoughts/Recommendations*: I do think we should migrate to the new
>>> > API.
>>> > > I
>>> > > > think the question is which of the new APIs we should use. The high
>>> > level
>>> > > > client seems to shield us from having to deal with constructing
>>> special
>>> > > > JSON handling code, whereas the low level client handles all
>>> versions
>>> > of
>>> > > > ES. I don't have a good feel (yet) for just how much work it would
>>> > > require
>>> > > > to use the low level API, or how difficult it would be to add new
>>> > request
>>> > > > features in the future. Actually, we could probably leverage
>>> existing
>>> > > code
>>> > > > we have for dealing with JSON maps, so this might be really easy.
>>> > Someone
>>> > > > with more experience in Metron's ES client use might have a better
>>> idea
>>> > > of
>>> > > > the pros and cons to this. The high level client appears to handle
>>> > > > everything all JSON manipulation for us, but we lose the benefit of
>>> a
>>> > > > simpler dependency tree and support for all versions of ES. My only
>>> > > concern
>>> > > > with "supports all versions" is that I have to imagine there are
>>> > specific
>>> > > > calls that we'd have to be careful of when constructing the JSON
>>> > > requests,
>>> > > > so it's unclear to me if this is better or worse in the end.
>>> > > >
>>> > > > Best,
>>> > > > Mike
>>> > > >
>>> > > >
>>> > > > 1. https://www.elastic.co/blog/state-of-the-official-
>>> > > > elasticsearch-java-clients
>>> > > > 2. https://www.elastic.co/guide/en/elasticsearch/client/java-
>>> > > > rest/current/java-rest-high-compatibility.html
>>> > > > <https://www.elastic.co/guide/en/elasticsearch/client/java-
>>> > > > rest/current/java-rest-high-compatibility.html>
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Wed, Sep 27, 2017 at 8:03 PM, Michael Miklavcic <
>>> > > > michael.miklav...@gmail.com> wrote:
>>> > > >
>>> > > > > I am working on upgrading Elasticsearch and Kibana. There are
>>> quite a
>>> > > few
>>> > > > > changes involved with this vix. I believe I'm mostly finished with
>>> > the
>>> > > > > Ambari mpack side of things, however we currently only support one
>>> > > > version
>>> > > > > with no backwards compatibility. What is the community's thoughts
>>> on
>>> > > > this?
>>> > > > >
>>> > > > > Here is some work contributed to the community that I'm
>>> referencing
>>> > > while
>>> > > > > working on this upgrade - https://github.com/apache/
>>> > > > metron/pull/619/files
>>> > > > >
>>> > > > > Best,
>>> > > > > Michael Miklavcic
>>> > > > >
>>> > > >
>>> > >
>>> >
---
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
Github user james-sirota commented on a diff in the pull request:
https://github.com/apache/metron/pull/788#discussion_r144174823
--- Diff:
metron-interface/metron-alerts/src/app/alerts/alert-details/alert-details.component.ts
---
@@ -133,6 +173,40 @@ export class
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/796
@ottobackwards @iraghumitra i already filed a feature request on that:
https://issues.apache.org/jira/browse/METRON-1248
---
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/796
I tested the PR. The only issue I see is that when I paste the timestamp
or manually type it into the boxes it overwrites it with the calendar entries.
So essentially the only way to get the
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/710
I can't comment on implementation details, but + 1 on the capabilities. I
spun it up and every time I click on a facet I see my search results filtered
by that facet. I can do that
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/787
+ 1 from me as well pending the completion of the Travis build
---
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/710
Ok, I opened
https://issues.apache.org/jira/browse/METRON-1250
as a follow on jira for this
---
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/579
Hi we would like to get this into the next release. @ctramnitz we'll be
happy to help you fix it
---
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/796
On my objection about not being able to paste a timestamp, I filed a
follow-on Jira so that this PR can go in
https://issues.apache.org/jira/browse/METRON-1253
---
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/803
You should not have empty meta alerts. That does not make sense
---
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/803
I filed the following follow-on PRs per your comments:
https://issues.apache.org/jira/browse/METRON-1268
https://issues.apache.org/jira/browse/METRON-1269
---
The Project Management Committee (PMC) for Apache Metron
has invited Raghu Mitra to become a committer and we are pleased
to announce that he has accepted.
Being a committer enables easier contribution to the
project since there is no need to go via the patch
submission process. This should en
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/811
@nickwallen what you are looking at is a desired behavior. If the alerts
are a part of the meta alert they do not appear in the facets
---
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/811
@nickwallen to avoid scope creep on this PR I created a follow-on PR to
figure out how to represent meta alerts in the facet panel.
https://issues.apache.org/jira/browse/METRON-1276
I
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/796
A few things didn't work for me. First, when I select a time range of (t-x
minutes) the start and end time does not fill in per screen shot below.
https:/
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/796
@iraghumitra looks like everything has been addressed. I am +1 on my side,
but lets have @merrimanr chime in
---
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/796
login to application
â should display error message for invalid credentials
â should login for valid credentials
â should logout
metron-alerts App
Github user james-sirota commented on the issue:
https://github.com/apache/metron/pull/796
+ 1. Gret job. all pass
login to application
â should display error message for invalid credentials
â should login for valid credentials
â should logout
1 - 100 of 102 matches
Mail list logo
