[GitHub] metron issue #879: METRON-1378: Create a summarizer

Github user justinleet commented on the issue: https://github.com/apache/metron/pull/879 @mmiklavc Check out https://github.com/apache/metron/pull/882#issuecomment-356109443. Looks like the squid mapping @cestella uses doesn't line up (which isn't terribly surprising because it was

[GitHub] metron issue #786: METRON-1231: Separate Sensor name and topic in the Manage...

Github user justinleet commented on the issue: https://github.com/apache/metron/pull/786 Spun it up, noticed one other problem. Unfortunately, again, I'm not sure if it's preexisting due to unfamiliarity. I stopped the bro topology (which was successful), then I started it again. I

[GitHub] metron pull request #879: METRON-1378: Create a summarizer

Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/879#discussion_r160179259 --- Diff: metron-platform/metron-data-management/README.md --- @@ -354,3 +357,91 @@ The parameters for the utility are as follows: | -r |

[GitHub] metron pull request #879: METRON-1378: Create a summarizer

Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/879#discussion_r160208209 --- Diff: metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/flatfile/writer/Writer.java --- @@ -0,0 +1,34 @@

[GitHub] metron pull request #879: METRON-1378: Create a summarizer

Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/879#discussion_r160177446 --- Diff: metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/flatfile/importer/AbstractLocalImporter.java --- @@

[GitHub] metron pull request #880: METRON-1379: Add an OBJECT_GET stellar function

Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/880#discussion_r160210633 --- Diff: metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/stellar/ObjectGetTest.java --- @@ -0,0 +1,91 @@ +/** + *

[GitHub] metron issue #690: METRON-1091 Package STELLAR shell as stand alone

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/690 This looks reasonable to me @ottobackwards. I'm +1 pending any further work per @JonZeolla's comments. Thanks for the contribution! ---

[GitHub] metron issue #785: METRON-1230: As a stopgap prior to METRON-777, add more s...

Github user cestella commented on the issue: https://github.com/apache/metron/pull/785 @ottobackwards One of the reasons why I did it the way that I did (hook into the way the storm accumulates the jar to submit) was that: 1. while it's useful for sideloading parsers, it's also

[GitHub] metron pull request #879: METRON-1378: Create a summarizer

GitHub user cestella reopened a pull request: https://github.com/apache/metron/pull/879 METRON-1378: Create a summarizer ## Contributor Comments We have a nice and generalized infrastructure for loading data into HBase and interacting with it via `flatfile_loader.sh` and

[GitHub] metron pull request #879: METRON-1378: Create a summarizer

Github user cestella closed the pull request at: https://github.com/apache/metron/pull/879 ---

[GitHub] metron issue #831: METRON-1302: Split up Indexing Topology into batch and ra...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/831 So, all of these parameters with the more generic names will be uniformly applicable to all possible backends? We will never have more than one of a given type at any time? ( s3/hdfs

[GitHub] metron pull request #891: METRON-1282 creating a new topic using the rest en...

GitHub user MohanDV opened a pull request: https://github.com/apache/metron/pull/891 METRON-1282 creating a new topic using the rest end point breaks the list topic API ## Contributor Comments Creating a kafka topic using the rest endpoint doesn't add the required ACL to

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/840 I ran this up in full dev again and verified the e2e tests now work similar to how they do in master. I also manually tested several other areas including the Alerts UI, Kibana and Swagger.

[GitHub] metron pull request #882: METRON-1380: Create a typosquatting use-case (comm...

Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/882#discussion_r160241987 --- Diff: use-cases/typosquat_detection/README.md --- @@ -0,0 +1,448 @@ + +# Problem Statement +

[GitHub] metron pull request #880: METRON-1379: Add an OBJECT_GET stellar function

Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/880#discussion_r160242210 --- Diff: metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/stellar/ObjectGetTest.java --- @@ -0,0 +1,91 @@ +/** + *

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

Github user justinleet commented on the issue: https://github.com/apache/metron/pull/840 At this point, I'm +1 since @merrimanr ran up the e2e tests. A couple people have put a fair amount of testing into this, and it seems like at this point we're at parity and not finding more

[GitHub] metron pull request #888: METRON-1389: Zeppelin notebook import does not wor...

Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/888#discussion_r160228422 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_master.py --- @@

[GitHub] metron pull request #840: METRON-939: Upgrade ElasticSearch and Kibana

Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/840 ---

[GitHub] metron issue #840: METRON-939: Upgrade ElasticSearch and Kibana

Github user cestella commented on the issue: https://github.com/apache/metron/pull/840 I want to pile on and give this my (non-binding since I contributed PRs against this PR) +1. LGTM! ---

[GitHub] metron pull request #879: METRON-1378: Create a summarizer

Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/879#discussion_r160239982 --- Diff: metron-platform/metron-data-management/README.md --- @@ -354,3 +357,91 @@ The parameters for the utility are as follows: | -r |

[GitHub] metron pull request #879: METRON-1378: Create a summarizer

Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/879#discussion_r160239950 --- Diff: metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/flatfile/writer/Writer.java --- @@ -0,0 +1,34 @@

[GitHub] metron pull request #882: METRON-1380: Create a typosquatting use-case (comm...

Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/882#discussion_r160245549 --- Diff: use-cases/typosquat_detection/README.md --- @@ -0,0 +1,448 @@ + +# Problem Statement +

[GitHub] metron issue #831: METRON-1302: Split up Indexing Topology into batch and ra...

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/831 Unless there are other things I should test, I think it looks pretty good. Data is coming through to ES + Kibana (with the new ES 5.6.2 code, no less!), the topologies appear as expected and

[GitHub] metron issue #789: METRON-1233: Remove description of Global configuration f...

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/789 Bump ---

[GitHub] metron pull request #863: METRON-1347: Indexing Topology should fail tuples ...

Github user mmiklavc commented on a diff in the pull request: https://github.com/apache/metron/pull/863#discussion_r160250862 --- Diff: metron-platform/metron-indexing/README.md --- @@ -15,6 +15,12 @@ Indices are written in batch and the batch size and batch timeout are specified

[GitHub] metron issue #831: METRON-1302: Split up Indexing Topology into batch and ra...

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/831 One additional note, I think we may be reaching the limits of what can reasonably fit on 1 box memory-wise. After about 40 minutes, the REST api and Elasticsearch went down. I recall this sort of

[GitHub] metron issue #831: METRON-1302: Split up Indexing Topology into batch and ra...

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/831 The code itself looks good to me. I'm running this up in full dev now and will report back. Fingers crossed that the extra slot is not a problem with the extra mem required for ES 5. ---

[GitHub] metron issue #786: METRON-1231: Separate Sensor name and topic in the Manage...

Github user justinleet commented on the issue: https://github.com/apache/metron/pull/786 Didn't see this happen while trying a second time time, or with snort and the logs don't seem to have anything interesting. ---

[GitHub] metron issue #873: METRON-1367 Stellar should have some instrumentation of f...

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/873 How is the timing handled for nested structures such as `STARTS_WITH(TO_UPPER(DECODE(...)))`? Is the timing for STARTS_WITH exclusive of TO_UPPER and DECODE or inclusive? ---

[GitHub] metron issue #872: METRON-1366: Add an entropy stellar function

Github user justinleet commented on the issue: https://github.com/apache/metron/pull/872 @cestella Any response to the comment by @ottobackwards ? I glanced over it, and I like it and think it's valuable, but he's hitting at the core impl, so I don't want to +1 anything. ---

[GitHub] metron issue #879: METRON-1378: Create a summarizer

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/879 @justinleet > barring the UI because of ES5 issues What sort of issues? ---

[GitHub] metron issue #873: METRON-1367 Stellar should have some instrumentation of f...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/873 Timing is integrated into the stellar execution, so it is taken before we execute named functions and lambdas. ---

[GitHub] metron issue #873: METRON-1367 Stellar should have some instrumentation of f...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/873 see the changes to LambdaExpression and StellarCompiler ---

[GitHub] metron issue #882: METRON-1380: Create a typosquatting use-case (commit afte...

Github user justinleet commented on the issue: https://github.com/apache/metron/pull/882 I ran through the instructions. The new data flowing automatically into the default ES mapping causes the problem that fielddata isn't true, so grouping queries don't match on the squid index

[GitHub] metron issue #879: METRON-1378: Create a summarizer

Github user justinleet commented on the issue: https://github.com/apache/metron/pull/879 I spun this up in the context of the combined PR, and everything worked as advertised, barring the UI because of ES5 issues. I was able to validate that data flowed through as expected by

[GitHub] metron issue #880: METRON-1379: Add an OBJECT_GET stellar function

Github user justinleet commented on the issue: https://github.com/apache/metron/pull/880 I ran this up in the combined PR, and it worked really well. As noted on that ticket, further changes are necessary in the instructions (to handle ES5), but for this ticket looking in ES for the

[GitHub] metron pull request #882: METRON-1380: Create a typosquatting use-case (comm...

Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/882#discussion_r160269501 --- Diff: use-cases/typosquat_detection/README.md --- @@ -0,0 +1,448 @@ + +# Problem Statement +

[GitHub] metron issue #882: METRON-1380: Create a typosquatting use-case (commit afte...

Github user cestella commented on the issue: https://github.com/apache/metron/pull/882 @justinleet Instructions updated, good catch. ---

[GitHub] metron pull request #886: METRON-1385: Missing "properties" in index templat...

Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/886 ---

[GitHub] metron pull request #878: METRON-1377: Stellar function to generate typosqua...

Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/878 ---

[GitHub] metron issue #786: METRON-1231: Separate Sensor name and topic in the Manage...

Github user justinleet commented on the issue: https://github.com/apache/metron/pull/786 @merrimanr You are absolutely right, my bad. Turns out I am illiterate in the morning. ---

[GitHub] metron pull request #831: METRON-1302: Split up Indexing Topology into batch...

Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/831#discussion_r160161491 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/hdfs.properties.j2 --- @@

[GitHub] metron pull request #831: METRON-1302: Split up Indexing Topology into batch...

Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/831#discussion_r160161476 --- Diff: metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/integration/IndexingIntegrationTest.java --- @@ -197,9 +140,7 @@ public

[GitHub] metron issue #831: METRON-1302: Split up Indexing Topology into batch and ra...

Github user cestella commented on the issue: https://github.com/apache/metron/pull/831 @nickwallen Whoops, sorry, I totally misunderstood. This PR *is* how *I* would like it. I'm just going to copy this here to justify my position: ``` The reason why I preferred to do that

[GitHub] metron issue #786: METRON-1231: Separate Sensor name and topic in the Manage...

Github user justinleet commented on the issue: https://github.com/apache/metron/pull/786 Yeah, 4 I'm definitely fine with being a separate PR. I'll spin this up again quick and take another look. ---

[GitHub] metron issue #786: METRON-1231: Separate Sensor name and topic in the Manage...

Github user justinleet commented on the issue: https://github.com/apache/metron/pull/786 @merrimanr Are those fixes the sort of thing we can/should add e2e tests for? I know those are flaky, but it seems like we should be able to have semi-automated confirmation on the fixes. ---

[GitHub] metron issue #786: METRON-1231: Separate Sensor name and topic in the Manage...

Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/786 We don't have e2e tests for the management UI. We do have unit tests and I added tests for these fixes there. ---

Full Dev -> Heartbeat issues

I just started up full dev from the 0.4.2 release tag, and ended up with failed heartbeats for all my services in ambari. After investigation, I found the my /etc/hosts ( on node1 ) had multiple entries for node1 : [vagrant@node1 ~]$ cat /etc/hosts 127.0.0.1 node1 node1 127.0.0.1 localhost ##

Re: Full Dev -> Heartbeat issues

I haven't seen that one. I spun one up from master on Friday and it seemed ok. Sorry, "works for me!" isn't super helpful, but it may be relevant since master is close to 0.4.2 :) On Mon, Jan 8, 2018 at 11:11 AM, Otto Fowler wrote: > I just started up full dev from