Github user cestella commented on the issue:
https://github.com/apache/metron/pull/922
Ugh, sorry, missed the guid field. *now* it's ready to test with guid
field set up.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/922
@merrimanr Things should be fixed at this point. I'm going to spin it up
and validate things this morning.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/922
@merrimanr Ah! Yes, we really should set a `guid` field for errors.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/915
@mraliagha Nah, they're not a troublemaker. I'm removing the PR since
there's interest in them staying turned on by default.
---
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/915
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/922
@merrimanr I'm confused about the guid field. I had to take that out given
the sample data that I generated from full-dev (they didn't have a `guid`
field). How did you create the errors?
---
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/924#discussion_r165982935
--- Diff:
metron-platform/metron-common/src/test/java/org/apache/metron/common/error/MetronErrorTest.java
---
@@ -53,7 +52,14 @@ public void
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/925
+1 by inspection
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/923
Looks like you didn't quite get all of the mock test infrastructure for
`metron-rest` set up properly. I went ahead and submitted a PR against your
branch to help out. :) If you can merge
https
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/922
Ok, I did the following:
* Augmented the readme to point to the Solr documentation around schemas.
Keep in mind, this is intermediate work that will feed into the "install Solr&q
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/922#discussion_r165711148
--- Diff:
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/schema/SchemaTranslatorTest.java
---
@@ -0,0 +1,188
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/922
@ottobackwards Very likely these schema files won't stay in this spot, but
the final resting spot won't be apparent until we figure out how to
automatically apply the schemas. Treat this PR
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/922#discussion_r165656636
--- Diff:
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/schema/SchemaTranslatorTest.java
---
@@ -0,0 +1,188
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/922#discussion_r165656511
--- Diff:
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/schema/SchemaTranslatorTest.java
---
@@ -0,0 +1,188
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/921
---
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/922
METRON-1441: Create complementary Solr schemas for the main sensors
## Contributor Comments
We have ES templates for bro, snort, yaf, and error, we need corresponding
solr schemas
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/911
actually, it may be that feature branch PRs don't automatically close..or
hmm, not sure. Anyway, close it please :)
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/911
well, I botched the stupid commit message and put `apache/metron` when I
meant `apache/feature/METRON-1416-upgrade-solr`. Can you close this PR
@merrimanr ?
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/921
I know that commit history looks pretty awful, but it's because I branched
off of @merrimanr 's SolrDao PR initially. Now that PR is in the branch, so
the comparison should be more sensible.
---
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/921
METRON-1441: Create complementary Solr schemas for the main sensors
## Contributor Comments
We have ES templates for bro, snort, yaf, and error, we need corresponding
solr schemas
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/919
so, I'm +1 on this and we probably want a discussion outside of this about
where to move `platform-info.sh`
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/917
+1 by inspection, great work!
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/915
@mraliagha Well, if it's being used explicitly and on an on-going manner,
then I'll remove this PR. Do you think there's some value to being able to
slice and dice in the global config the ability
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/907
@nickwallen Awesome, your PR is merged in here and I'll spin it up one last
time to verify that it works for 2.0.0.2.
@anandsubbu Awesome, thanks!
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/915
Whoops, missed one. Thanks @mmiklavc !
---
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/915
METRON-1433: Only emit debugging timing fields in enrichment when debugging
is turned on
## Contributor Comments
Right now we always emit performance debugging fields in the split/join
bolts
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/907
For 2, that's the same error @mmiklavc hit on centos. It's very
interesting that I'm not hitting it.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/903
Looks good to me!
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/907
ok, I ran this guy up in kerberos and regular and tooled around a bit;
ensured the alerts UI worked in both.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/903
Honestly my only beef with this is the directory naming. I think I'd
prefer `metron-deployment/vagrant/{centos_$VERSION,ubuntu_$VERSION}` because
it's possible that we may want 2 separate versions
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/910#discussion_r163967410
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/IndexDao.java
---
@@ -140,9 +147,9 @@ default Document
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/910
Yeah, this is the first step to moving it out, I'd say. This was useful
independent of #907 because it made some things a lot cleaner (namely fewer
`new TypeReference<Map<String,
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/910
METRON-1430: Isolate jackson from being used as arguments or returns from
JSONUtils
## Contributor Comments
Currently jackson is used as part of our internal API to JSONUtils. The
problem
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/907
METRON-1427: Add support for storm 1.1 and hdp 2.6
## Contributor Comments
Right now our ambari mpack won't run cleanly on HDP 2.6 and Storm 1.1
because of some classpath issues
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/907
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/909
Sounds good, +1 Good work here, that test was confusing.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/909
Should this be against master or should this be committed against the Solr
branch? It *seems* like this is general purpose goodness and maybe fits in
master, but I wanted to double check.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/906
+1
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/908
+1 lgtm
---
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/901#discussion_r163607202
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/907#discussion_r163606292
--- Diff: metron-deployment/roles/ambari_config/vars/single_node_vm.yml ---
@@ -87,6 +87,11 @@ configurations:
supervisor.slots.ports: "
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/907#discussion_r163602345
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.2/package/scripts/service_check.py
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/907
METRON-1427: Add support for storm 1.1 and hdp 2.6
## Contributor Comments
Right now our ambari mpack won't run cleanly on HDP 2.6 and Storm 1.1
because of some classpath issues
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/905
+1, anything to save room on vagrant at the moment.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/904
+1 by inspection, thanks!
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/853
Just ducking in here, @merrimanr is this ready for review? Specifically,
the responses to @justinleet 's questions have all been factored into the
current code for this PR, right? If yes
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/687
So, I was wrong when I initially looked at this PR, this adds variable
updates to the resolver. Should we perhaps separate parsing assignments (e.g.
move `StellarAssignment.from` into the language
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/687
@nickwallen Yes, you are absolutely correct. Right now we do not have an
approach to take a list of stellar expressions that are assignments, a variable
resolver, a function resolver and execute
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/687
@nickwallen No, both the REPL and enrichments use the same method of
assignment, the `StellarAssignment` class. It's just that assignment in
stellar is available in multiple places.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/687
@nickwallen No, that's not correct. Stellar enrichments can contain `:=`.
Consider the enrichments
[here](https://github.com/apache/metron/tree/master/use-cases/geographic_login_outliers#enrich
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/687
hah, well @nickwallen, laziness is often a good trait in a developer.
Honestly, I am ok with *not* migrating here in favor of more intuitive
approaches. I chose `:=` because I needed
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/687
Yeah, I'm totally cool with going with `=`, but I will point out one snag.
What about users with existing stellar enrichments using `:=`, would we provide
a migration path beyond saying, &quo
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/894
I'm +1 on this at this point, but suggest we wait until @anandsubbu gives
his +1.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/687
Agreed, this is a POV issue. Though from the user perspective they aren't
distinguishing between "this is in the REPL" vs "this is in the enrichment
topology", so I think
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/687
well, `:=` is in the shell and also in the enrichment configuration (see
[here](https://github.com/apache/metron/tree/master/metron-platform/metron-enrichment#stellar-enrichment-configuration
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/879
haha @ottobackwards neutral would be a +0, which is fine. Thanks for your
constructive comments on the discuss thread and here. As always, they're much
appreciated. :)
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/879
Ok, @justinleet has given a +1, do we have any existing reservations after
the discussion thread and the review here on this work? If not, then I'm going
to commit on Monday.
---
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/856#discussion_r161240685
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
---
@@ -0,0
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/888#discussion_r161239480
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/863#discussion_r161087829
--- Diff: metron-platform/metron-indexing/README.md ---
@@ -15,6 +15,12 @@ Indices are written in batch and the batch size and
batch timeout are specified
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/872#discussion_r160532240
--- Diff:
metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/informationtheory/InformationTheoryUtil.java
---
@@ -0,0 +1,52
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/872
@justinleet I think that was just an alternative implementation with
streams rather than explicit loops. I interpreted it as not a suggested
change, but a cool new way to compute entropy.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/893
+1, ran it up, tried it out with the new template and both queries worked
as expected.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/879
Just following up, I have migrated the mapping of existing data to a
template in the instructions and the type mismatch for `ip_dst_addr` is no
longer an issue. The PR as it currently stands has
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/882
@justinleet Instructions updated, good catch.
---
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/882#discussion_r160269501
--- Diff: use-cases/typosquat_detection/README.md ---
@@ -0,0 +1,448 @@
+
+# Problem Statement
+
+[Typosquatting](https://en.wikipedia.org
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/880#discussion_r160242210
--- Diff:
metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/stellar/ObjectGetTest.java
---
@@ -0,0 +1,91
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/879#discussion_r160239982
--- Diff: metron-platform/metron-data-management/README.md ---
@@ -354,3 +357,91 @@ The parameters for the utility are as follows:
| -r
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/879#discussion_r160239950
--- Diff:
metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/flatfile/writer/Writer.java
---
@@ -0,0 +1,34
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/840
I want to pile on and give this my (non-binding since I contributed PRs
against this PR) +1. LGTM!
---
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/888#discussion_r160228422
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/879
---
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/879
METRON-1378: Create a summarizer
## Contributor Comments
We have a nice and generalized infrastructure for loading data into HBase
and interacting with it via `flatfile_loader.sh
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/785
@ottobackwards One of the reasons why I did it the way that I did (hook
into the way the storm accumulates the jar to submit) was that:
1. while it's useful for sideloading parsers, it's also
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/831
@nickwallen Whoops, sorry, I totally misunderstood. This PR *is* how *I*
would like it. I'm just going to copy this here to justify my position:
```
The reason why I preferred to do
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/831#discussion_r160161491
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/hdfs.properties.j2
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/831#discussion_r160161476
--- Diff:
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/integration/IndexingIntegrationTest.java
---
@@ -197,9 +140,7 @@ public
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/879
So, the discuss thread has been going for some time now and the discussion
is mostly around forward-thinking extensions to this. Are we at the point to
agree that this is a viable first step
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/878
Ok, I added better comments around the various strategies. Let me know if
you see anything else.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/831
Ok, I thought the general consensus was different. As per my comment:
```
That sounds good. Which one would you guys prefer:
* display names in the mpack of "Elastics
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/887
+1 by inspection
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/879
After more consideration and more egg nog, I decided that I'd create a
DISCUSS thread about this entire use-case. We can move the discussion there.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/879
Also, a wizard-like UI could simplify this dramatically. That was one of
the thoughts around extending and reusing the existing infrastructure in the
first pass of this rather than creating a new
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/879
@ottobackwards Yes, that's spot on. It's to enable creation of
summarization objects in a method similar (and reusing the configs and
infrastructure of) the flat file loader. The idea
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/879
@nickwallen I definitely hear you, the JSON configs are more complex than
I'd like. I'd like another more composable solution available using lambda
functions available in the REPL
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/882
METRON-1380: Create a typosquatting use-case (commit after METRON-1379,
METRON-1377, METRON-1378)
## Contributor Comments
This is a documented use-case on how to use the following JIRAs (PRs
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/882
---
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/882
METRON-1380: Create a typosquatting use-case
## Contributor Comments
This is a documented use-case on how to use the following JIRAs (PRs) to
detect typosquatting in-stream using bloom filters
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/880
METRON-1379: Add an OBJECT_GET stellar function
## Contributor Comments
With the creation of METRON-1378 we have the ability to create serialized
summaries of data. We need to interact
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/879
METRON-1378: Create a summarizer
## Contributor Comments
We have a nice and generalized infrastructure for loading data into HBase
and interacting with it via `flatfile_loader.sh
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/871
Thanks for the review, @nickwallen I reacted to your comments. :)
---
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/871#discussion_r158040472
--- Diff:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerClientConfig.java
---
@@ -56,8 +56,11
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/871#discussion_r158040446
--- Diff:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
---
@@ -39,11 +39,7
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/871#discussion_r158040425
--- Diff:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
---
@@ -310,6 +306,36
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/870
Yep, I owe a manual testing plan. It's going to essentially be the same
instructions as we have for
[MAD](https://github.com/apache/metron/tree/master/metron-analytics/metron-statistics#median
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/878
METRON-1377: Stellar function to generate typosquatted domains (similar to
dnstwist)
## Contributor Comments
As a component of a strategy to detect [Typosquatting](
https://en.wikipedia.org
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/875
Is it sensible to exclude `dist/styles.*.bundles.css`? It seems that this
file changes its name.
---
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/872
METRON-1366: Add an entropy stellar function
## Contributor Comments
Trending entropy for various volumetric statistics (e.g. netflow data) has
been a useful metric for intrusion detection
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/871
METRON-1365: Allow PROFILE_GET to return a default value for a profile and
entity that does not have a value written.
## Contributor Comments
Right now the profiler is a sparse system, namely
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/870
METRON-1364: Add an implementation of Robust PCA outlier detection
## Contributor Comments
With short circuiting in Stellar, we have the opportunity to delve into
more computationally
101 - 200 of 729 matches
Mail list logo
