Re: Metron Alert UI and zero-down time Elasticsearch re-index

Hi James, Due to changes in the field format, I want to create a new index with the new format. Create an alias to refer to both new and old index. Then, copy all the documents from the old index to the new index and use the alias to search through Metron Alert UI and Kibana to avoid any

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/856#discussion_r160002156 --- Diff: metron-platform/metron-management/README.md --- @@ -276,6 +278,13 @@ The functions are split roughly into a few sections: *

[GitHub] metron issue #884: METRON-1382 Run Stellar in a Zeppelin Notebook

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/884 > Also, do we now have to validate in both environments when we make changes to the shell? that should also be spelt out. No you do not have to validate in both. The point of all this

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/856#discussion_r160001387 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java --- @@

[GitHub] metron issue #884: METRON-1382 Run Stellar in a Zeppelin Notebook

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/884 > When I say I would like a readme or outline document, here is where I'm coming from: As someone who does do stellar and stellar shell work, I would like to understand the high level view of

[GitHub] metron pull request #884: METRON-1382 Run Stellar in a Zeppelin Notebook

Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/884#discussion_r16380 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/DefaultStellarShellExecutor.java --- @@ -0,0 +1,413 @@

[GitHub] metron pull request #884: METRON-1382 Run Stellar in a Zeppelin Notebook

Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/884#discussion_r16011 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShellExecutor.java --- @@ -0,0 +1,79 @@ +/*

[GitHub] metron pull request #884: METRON-1382 Run Stellar in a Zeppelin Notebook

Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/884#discussion_r15880 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarResult.java --- @@ -0,0 +1,185 @@ +/* + *

[GitHub] metron pull request #884: METRON-1382 Run Stellar in a Zeppelin Notebook

Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/884#discussion_r15794 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarExecutionListeners.java --- @@ -0,0 +1,51 @@ +/*

[GitHub] metron pull request #884: METRON-1382 Run Stellar in a Zeppelin Notebook

Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/884#discussion_r15558 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarResult.java --- @@ -0,0 +1,185 @@ +/* + *

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/856#discussion_r15157 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java --- @@ -0,0

[GitHub] metron issue #786: METRON-1231: Separate Sensor name and topic in the Manage...

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/786 @justinleet 1, 2, and 3 have been addressed with the latest commit. I also added a couple unit tests for good measure. I wasn't able to easily reproduce 4 so I didn't fix that one. It

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/840 Worked with @merrimanr to fix the e2e test issue and just submitted a fix. This does not fix the full e2e test runs as this is being handled by 857, however it brings this PR back to parity with

[GitHub] metron issue #884: METRON-1382 Run Stellar in a Zeppelin Notebook

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/884 When I say I would like a readme or outline document, here is where I'm coming from: As someone who does do stellar and stellar shell work, I would like to understand the high level view of

[GitHub] metron issue #878: METRON-1377: Stellar function to generate typosquatted do...

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/878 Nice work, +1 ---

[GitHub] metron issue #884: METRON-1382 Run Stellar in a Zeppelin Notebook

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/884 I would not, and am not -1 a pr for non-technical reasons. I'm not ready to +1 this, since I have outstanding comments and actually have to try it out beyond scanning the diffs. ---

Re: [DISCUSS] Generating and Interacting with serialized summary objects

Any volunteers for creating a set of jiras and feature branch for an object store repository? This sounds like a massive feature. On Jan 5, 2018 2:06 PM, "Otto Fowler" wrote: > I would say that at the stellar author level, you would just get objects > from the store and

Re: [DISCUSS] Generating and Interacting with serialized summary objects

Yes, abstracted. We have an api of stellar functions that just load things from the store, they don’t need to bleed through what the store is. We have a ‘store’, which may be hdfs or hbase or whatever. We have an api for adding to the store ( add etc ) that doesn’t presume the store either. Then

Re: [DISCUSS] Generating and Interacting with serialized summary objects

I would say that at the stellar author level, you would just get objects from the store and the ‘override’ case would be a follow on for edge cases. On January 5, 2018 at 14:29:16, Casey Stella (ceste...@gmail.com) wrote: Well, you can pull the default configs from global configs, but you might

[GitHub] metron issue #879: METRON-1378: Create a summarizer

Github user cestella commented on the issue: https://github.com/apache/metron/pull/879 So, the discuss thread has been going for some time now and the discussion is mostly around forward-thinking extensions to this. Are we at the point to agree that this is a viable first step and

[GitHub] metron issue #878: METRON-1377: Stellar function to generate typosquatted do...

Github user cestella commented on the issue: https://github.com/apache/metron/pull/878 Ok, I added better comments around the various strategies. Let me know if you see anything else. ---

Re: [DISCUSS] Generating and Interacting with serialized summary objects

Well, you can pull the default configs from global configs, but you might want to override them (similar to the profiler). For instance, you might want to interact with another hbase table than the one globally configured. On Fri, Jan 5, 2018 at 12:04 PM, Otto Fowler

Re: [DISCUSS] Generating and Interacting with serialized summary objects

I'm not sure I follow what you're saying as it pertains to summary objects. Repository is a loaded term, and I'm very apprehensive of pushing for something potentially very complex where a simpler solution would suffice in the short term. To wit, the items I'm seeing in this use case doc -

[GitHub] metron issue #884: METRON-1382 Run Stellar in a Zeppelin Notebook

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/884 > Overall: I would like to see this wait until we make some effort to clear outstanding stellar PR's from the queue. Making PR authors whose PR's don't get reviewed continually update to

Re: [DISCUSS] Generating and Interacting with serialized summary objects

I would imagine the ‘stellar-object-repo’ would be part of the global configuration or configuration passed to the command. why specify in the function itself? On January 5, 2018 at 11:22:32, Casey Stella (ceste...@gmail.com) wrote: I like that, specifically the repositories abstraction.

[GitHub] metron pull request #884: METRON-1382 Run Stellar in a Zeppelin Notebook

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/884#discussion_r159921114 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarResult.java --- @@ -0,0 +1,185 @@ +/* + *

[GitHub] metron pull request #884: METRON-1382 Run Stellar in a Zeppelin Notebook

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/884#discussion_r159922114 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShellExecutor.java --- @@ -0,0 +1,79 @@ +/*

[GitHub] metron pull request #884: METRON-1382 Run Stellar in a Zeppelin Notebook

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/884#discussion_r159920465 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarResult.java --- @@ -0,0 +1,185 @@ +/* + *

[GitHub] metron pull request #884: METRON-1382 Run Stellar in a Zeppelin Notebook

Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/884#discussion_r159919435 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/DefaultStellarShellExecutor.java --- @@ -0,0 +1,413 @@

Re: [DISCUSS] Generating and Interacting with serialized summary objects

I like that, specifically the repositories abstraction. Perhaps we can construct some longer term JIRAs for extensions. For the current state of affairs (wrt to the OBJECT_GET call) I was imagining the simple default HDFS solution as a first cut and following on adding a repository name (e.g.

[GitHub] metron issue #884: METRON-1382 Run Stellar in a Zeppelin Notebook

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/884 I spun this up again and all the issues I found have been resolved. Great work. +1 ---

Re: [DISCUSS] Generating and Interacting with serialized summary objects

If we separate the concerns as I have state previously : 1. Stellar can load objects into ‘caches’ from some repository and refer to them. 2. The repositories 3. Some number of strategies to populate and possibly update the repository, from spark, to MR jobs to whatever you would classify the

[GitHub] metron issue #831: METRON-1302: Split up Indexing Topology into batch and ra...

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/831 > @cestella Nobody commented on that, so I went with adjusting the display names rather than variable prefixes... I think there is some confusion as to what exactly you mean here. You

notices_dumper.py

/notices_dumper.py Do we need this anymore? There are no docs in it and I can't find any references to it anywhere else in the code base. Based on when it was committed, it might have done something with the legacy OpenSOC UI code that is no longer with us.

[GitHub] metron issue #786: METRON-1231: Separate Sensor name and topic in the Manage...

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/786 I believe 2 and 4 are preexisting but I'm happy to fix them here. Will update when I have resolved these issues. ---

Re: [DISCUSS] Generating and Interacting with serialized summary objects

I agree with the general sentiment that we can tailor specific use cases via UI, and I'm worried that the use case specific solution (particularly in light of the note that it's not even general to the class of bloom filter problems, let alone an actually general problem) becomes more work than

[GitHub] metron pull request #889: METRON-1390: Swagger UI for "Web Security Config" ...

GitHub user MohanDV reopened a pull request: https://github.com/apache/metron/pull/889 METRON-1390: Swagger UI for "Web Security Config" Controller needs request method ## Contributor Comments Swagger UI for "Web Security Config" Controller needs request method to be

[GitHub] metron pull request #889: METRON-1390: Swagger UI for "Web Security Config" ...

Github user MohanDV closed the pull request at: https://github.com/apache/metron/pull/889 ---

[GitHub] metron issue #786: METRON-1231: Separate Sensor name and topic in the Manage...

Github user justinleet commented on the issue: https://github.com/apache/metron/pull/786 I took a swing through this, and generally this worked as expected. There were a couple things in full dev that I'm not sure if they were preexisting or not, just from unfamiliarity with how it

[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana

Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/840#discussion_r159866359 --- Diff: pom.xml --- @@ -159,7 +159,7 @@ ${global_surefire_version}

[GitHub] metron pull request #889: METRON-1390: wagger UI for "Web Security Config" C...

GitHub user MohanDV opened a pull request: https://github.com/apache/metron/pull/889 METRON-1390: wagger UI for "Web Security Config" Controller needs request method ## Contributor Comments Swagger UI for "Web Security Config" Controller needs request method to be added