Github user cestella commented on the issue:
https://github.com/apache/metron/pull/966
+1 by inspection.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/914
I love this, sorry I didn't review it sooner @ottobackwards +1 by
inspection. This is great.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/959
+1 by inspection, thanks Jon!
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/964
Ok, created
[METRON-1492](https://issues.apache.org/jira/browse/METRON-1492).
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/964
@ottobackwards yeah, definitely; I think that's ultimately where we want to
go. The first step to that is separating out these functions like I have in
this PR. The next is doing the ambari work
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/964
METRON-1491: The indexing topology restart logic is wrong
## Contributor Comments
If either topology is down, Ambari shows all of Indexing as dead. Clicking
start attempts to start them both
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174787158
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/SendToKafka.java
---
@@ -0,0 +1,107
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/958
Moving this conversation to the top. I believe I have refactored the
KafkaProducers appropriately. Let me know if I missed something, @justinleet
---
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/963
METRON-1490: Better error message when user specifies an enrichment type
that doesn't exist
## Contributor Comments
If a user specifies an enrichment adapter name that doesn't exist (e.g
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174478516
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/SendToKafka.java
---
@@ -0,0 +1,107
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/942
So, I would suggest rather than accepting a list of stats objects, `MAX`
and `MIN` accept one of the following:
* A StatisticsProvider object
* A list of comparables
Essentially
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174186826
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/sampler/BiasedSampler.java
---
@@ -0,0 +1,95
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174186723
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/sampler/BiasedSampler.java
---
@@ -0,0 +1,95
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174186700
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/monitor/writers/Writer.java
---
@@ -0,0 +1,91
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174186800
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/sampler/UnbiasedSampler.java
---
@@ -0,0 +1,28
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174186641
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/monitor/writers/Writer.java
---
@@ -0,0 +1,91
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174186597
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/monitor/writers/ConsoleWriter.java
---
@@ -0,0 +1,67
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174186529
--- Diff:
metron-contrib/metron-performance/src/test/java/org/apache/metron/performance/load/SendToKafkaTest.java
---
@@ -0,0 +1,49
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174186458
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadOptions.java
---
@@ -0,0 +1,504
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174186359
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadGenerator.java
---
@@ -0,0 +1,165
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174186403
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadGenerator.java
---
@@ -0,0 +1,165
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174186422
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadOptions.java
---
@@ -0,0 +1,504
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174186490
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/MessageGenerator.java
---
@@ -0,0 +1,48
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174186316
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadGenerator.java
---
@@ -0,0 +1,165
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174186275
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadGenerator.java
---
@@ -0,0 +1,165
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/958
@justinleet thanks for the review. I reacted to your comments either by
fixing them or suggesting why I prefer what is there. I will add a new set of
tests in the next commit.
---
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r174181759
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/SendToKafka.java
---
@@ -0,0 +1,107
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/958#discussion_r173942355
--- Diff:
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/monitor/AbstractMonitor.java
---
@@ -0,0 +1,49
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/961#discussion_r173928377
--- Diff: metron-platform/metron-enrichment/Performance.md ---
@@ -0,0 +1,522 @@
+
+
+# Enrichment Performance
+
+This guide defines
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/962
METRON-1488: user_settings hbase table does not have acls set up for
kerberos
## Contributor Comments
Currently some REST calls will fail because we do not set ACL's on the new
user_settings
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/958
# Testing Plan:
We presume
* `ZOOKEEPER` is an environment variable set to the zk quorum (e.g.
`node1:2181`)
* `BROKER` is an environment variable set to the broker (e.g. `node1:6667
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/949
---
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/949
METRON-1471: Migrate shuffle connections to local or shuffle
## Contributor Comments
Currently, we use shuffle groupings when we do not want to group by field.
We should, instead, use local
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/958
METRON-1483: In performance evaluation, generating synthetic load and
monitoring the write throughput of our kafka-to-kafka topologies has required a
lot of custom scripting. We should have a tool
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/945
Ok, I'm cool with it. +1 by inspection; great work.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/949
@merrimanr Ah, I had forgotten to merge in master yesterday afternoon.
There should be no more shuffles in any of the topologies.
---
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/947#discussion_r172870992
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java
---
@@ -89,29 +91,25 @@ public void prepare(Map
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/945
@merrimanr you ran this up in Solr 5.5 as well as 6.6, right? If so, then
I'm content with the change and give a +1 pending (other than holding for an
answer to Simon's question, which I
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/952#discussion_r172858774
--- Diff: metron-interface/metron-alerts/package-lock.json ---
@@ -1,6427 +0,0 @@
-{
--- End diff --
I suspect the build failure is due
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/940#discussion_r172711543
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java
---
@@ -0,0 +1,281
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/940#discussion_r172656809
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java
---
@@ -0,0 +1,281
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/949
METRON-1471: Migrate shuffle connections to local or shuffle
## Contributor Comments
Currently, we use shuffle groupings when we do not want to group by field.
We should, instead, use local
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
Ok, README is updated with the new topology diagram. Let me know if
there's anything else.
---
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/940#discussion_r172383791
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/Strategy.java
---
@@ -0,0 +1,47
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/940#discussion_r172383810
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java
---
@@ -0,0 +1,415
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
@nickwallen Ok, I refactored the abstraction to separate some concerns,
name things a bit, and collapse some of the more onerous abstractions. Also
updated javadocs.
Can you give
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/940#discussion_r172383754
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/EnrichmentStrategies.java
---
@@ -0,0 +1,79
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/940#discussion_r172377136
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/EnrichmentStrategies.java
---
@@ -0,0 +1,79
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/940#discussion_r172373203
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java
---
@@ -0,0 +1,415
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/940#discussion_r172369461
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/EnrichmentStrategies.java
---
@@ -0,0 +1,79
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/940#discussion_r172369480
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/Strategy.java
---
@@ -0,0 +1,47
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
Ahhh, that makes sense. I bet we were getting killed by small allocations
in the caching layer.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
I actually suspect GC as well. We adjusted the garbage collector to the
G1GC and saw throughput gains, but not nearly the kinds of gains as we got with
a drop-in of Caffeine to replace Guava.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
In this case, the loader isn't doing anything terribly expensive, though it
may in the future (incur a hbase get or some more expensive computation).
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/924
+1, sorry!
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
We actually did increase the concurrency level for guava to 64; that is
what confused us as well. The hash code is mostly standard, should be evenly
distributed (the key is pretty much a POJO).
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
The interesting thing that we found was that guava seems to be doing poorly
when the # of items in the cache gets large. When we scaled the test down (830
distinct IP addresses chosen randomly
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
We were being purposefully unkind to the cache in the tests. The load
simulation chose a IP address at random to present, so each IP had an equal
probability of being selected. Whereas, in real
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
I ran this up with vagrant and ensured:
* Normal stellar works still in field transformations as well as enrichments
* swapped in and out new enrichments live
* swapped in and out new
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/944
I ran this up with vagrant and ensured:
* Normal stellar works still in field transformations as well as enrichments
* swapped in and out new enrichments live
* swapped in and out new
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/947
I ran this up with vagrant and ensured:
* Normal stellar works still in field transformations as well as enrichments
* swapped in and out new enrichments live
* swapped in and out new
GitHub user cestella reopened a pull request:
https://github.com/apache/metron/pull/947
METRON-1467: Replace guava caches in places where the keyspace might be
large (NOTE: Review after METRON-1460)
## Contributor Comments
Based on the performance tuning exercise as part
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/947
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
Just FYI, as part of the performance experimentation in the lab here, we
found that one major impediment to scale was the guava cache in this topology
when the size of the cache becomes non-trivial
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/947
METRON-1467: Replace guava caches in places where the keyspace might be
large
## Contributor Comments
Based on the performance tuning exercise as part of METRON-1460, guava has
difficulties
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
@arunmahadevan Thanks for chiming in Arun. I would say that most of the
enrichment work is I/O bound and we try to avoid it whenever possible with a a
time-evicted LRU cache in front
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
@ottobackwards I haven't sent an email to the storm team, but I did run the
PR past a storm committer that I know and asked his opinion prior to submitting
the PR. The general answer was something
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
@mraliagha It's definitely a tradeoff. This is why this is as a complement
to the original split/join topology. Keep in mind, also, that this
architecture enables use-cases that the other would
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/946
ah, crap, looked at the wrong setting. That's what I meant instead of
`storm.library.path`
---
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/946#discussion_r171624949
--- Diff: metron-platform/elasticsearch-shaded/pom.xml ---
@@ -31,7 +43,7 @@
org.elasticsearch.client
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/946
@mmiklavc Yeah, I think that's the approach, however, there's a snag.
Storm requires us to create uber jars, so probably what we want to do is have
users actually put the xpath transport client
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/946#discussion_r171618015
--- Diff: metron-platform/elasticsearch-shaded/pom.xml ---
@@ -31,7 +43,7 @@
org.elasticsearch.client
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
@nickwallen Sounds good. When scale tests are done, can we make sure that
we also include #944 ?
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/938
+1 as well, looks great
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/938
Where is that powered by apache logo from? Are we sure it doesn't mean
that the apache web server serves it up?
---
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/944
METRON-1463: Adjust the groupings and shuffles in enrichment to be more
efficient
## Contributor Comments
Currently there are some deficiencies in our grouping approach in the
enrichment
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/934
+1 by inspection
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/940
The current architecture is described 
In short, for each message each
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/933
piling on, +1 by inspection
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/934
Chiming in late. I agree that we should not have an explicit dependency on
an indexing Mpack, even one not of our own construction. I think people will
have a lot of different ways to install
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/934#discussion_r170269639
--- Diff: metron-platform/metron-solr/src/main/scripts/create_collection.sh
---
@@ -0,0 +1,27 @@
+#!/bin/bash
+#
+# Licensed to the Apache
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/940#discussion_r170089790
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java
---
@@ -0,0 +1,323
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/940#discussion_r170057327
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java
---
@@ -0,0 +1,323
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/940
Single bolt split join poc
## Contributor Comments
There are some deficiencies to the split/join topology.
It's hard to reason about
* Understanding the latency of enriching
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/935
This looks great! +1 by inspection.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/937
I verified this in full dev with kerberos as well.
---
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/937
METRON-1455: Patch and Replace methods in the REST UpdateController return
400
## Contributor Comments
Shading and relocating the jackson library had the unexpected behavior of
breaking patch
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/929
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/929
I updated the instructions and the `solr.zookeeper` business is legacy
naming, but I think it does fit since it's specific to zookeeper rather than
connecting to ES master. I dunno, that's just
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/929
Ok, I wrote up a detailed testing plan to get this running in full-dev. I
want to add a couple of unit tests around the new properties that I added, but
at that point, this is feature complete
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/929#discussion_r166798136
--- Diff:
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/writer/SolrWriter.java
---
@@ -33,17 +39,19 @@
import
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/929
Ok, I updated the configs to expose all the settings that I could find for
committing and documented them. By default we operate safely, though. Though
now you can use soft commits rather than
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/929
LOL, nah, I was just being crotchety. Let me see what I can do and you
tell me what you think :)
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/929
Actually, I can make an option to let you do a soft commit instead of a
hard commit if that'll be an extra layer of configuration.
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/929
You can set it to not manually commit (set `solr.commitPerBatch` to `false`
and no committing happens), but you're risking losing data if a worker dies.
Honestly, want a durable commit
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/929
What do you mean "auto-commit"? By default it will commit at the batch
level as the other writers do. You can turn that behavior off and it'll commit
when the Solr client decides a
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/929#discussion_r166682417
--- Diff:
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/writer/SolrWriter.java
---
@@ -33,17 +39,19 @@
import
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/929
METRON-1448: Update SolrWriter to conform to new collection strategy
## Contributor Comments
Currently the SolrWriter presumes a single collection to be written to.
The new collection
Github user cestella closed the pull request at:
https://github.com/apache/metron/pull/922
---
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/922
@simonellistonball Yes, we should. I added the relevant context and
grouping for each of the schemas that we ship by default. Bro has more context
as there were more comments in the ES schema. I
1 - 100 of 729 matches
Mail list logo
