[GitHub] metron issue #966: METRON-1493 Unhelpful Error Message When Assignment Expre...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/966 +1 by inspection. ---
[GitHub] metron issue #914: METRON-1397 Support for JSON Path and complex documents i...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/914 I love this, sorry I didn't review it sooner @ottobackwards +1 by inspection. This is great. ---
[GitHub] metron issue #959: METRON-1485 Upgrade vagrant for dev environments
Github user cestella commented on the issue: https://github.com/apache/metron/pull/959 +1 by inspection, thanks Jon! ---
[GitHub] metron issue #964: METRON-1491: The indexing topology restart logic is wrong
Github user cestella commented on the issue: https://github.com/apache/metron/pull/964 Ok, created [METRON-1492](https://issues.apache.org/jira/browse/METRON-1492). ---
[GitHub] metron issue #964: METRON-1491: The indexing topology restart logic is wrong
Github user cestella commented on the issue: https://github.com/apache/metron/pull/964 @ottobackwards yeah, definitely; I think that's ultimately where we want to go. The first step to that is separating out these functions like I have in this PR. The next is doing the ambari work which will utilize it. Shall I create a follow-on JIRA? ---
[GitHub] metron pull request #964: METRON-1491: The indexing topology restart logic i...
GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/964 METRON-1491: The indexing topology restart logic is wrong ## Contributor Comments If either topology is down, Ambari shows all of Indexing as dead. Clicking start attempts to start them both and fails if either is still running. Furthermore, it appears to retry 3 times before finally failing the command. In order to test this, kill one of the indexing topologies then attempt to start indexing from ambari. Things should go smoothly. Also, try restarting and starting in various states. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron METRON-1491 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/964.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #964 commit 423e9a074492ae2071022b9a4adb347d2a109633 Author: cstella <cestella@...> Date: 2018-03-15T14:27:09Z METRON-1491: The indexing topology restart logic is wrong ---
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174787158 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/SendToKafka.java --- @@ -0,0 +1,107 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.load; + +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.ProducerRecord; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.TimerTask; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Future; +import java.util.concurrent.atomic.AtomicLong; +import java.util.function.Supplier; + +public class SendToKafka extends TimerTask { --- End diff -- alright, done and done. ---
[GitHub] metron issue #958: METRON-1483: Create a tool to monitor performance of the ...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/958 Moving this conversation to the top. I believe I have refactored the KafkaProducers appropriately. Let me know if I missed something, @justinleet ---
[GitHub] metron pull request #963: METRON-1490: Better error message when user specif...
GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/963 METRON-1490: Better error message when user specifies an enrichment type that doesn't exist ## Contributor Comments If a user specifies an enrichment adapter name that doesn't exist (e.g. `hbaseEnrichment` vs `hbaseThreatIntel`), then we NPE rather than express the issue in the logs. To test this, try to create an enrichment with an incorrect name. For instance, an enrichment like so: ``` { "enrichment": { "fieldMap": {}, "fieldToTypeMap": {}, "config": {} }, "threatIntel": { "fieldMap": { "hbaseEnrichment": ["ip_src_addr", "ip_dst_addr"] }, "fieldToTypeMap": { "ip_src_addr": ["malicious_ip"], "ip_dst_addr": ["malicious_ip"] }, "config": {}, "triageConfig": { "riskLevelRules": [], "aggregator": "MAX", "aggregationConfig": {} } }, "configuration": {} } ``` Because there is no enrichment adapter named `hbaseEnrichment` available for threat intel (the correct name is `hbaseThreatIntel`), an error should be noted in the logs and there should be a sensible message, such as: `java.lang.IllegalStateException: Unable to find an adapter for hbaseEnrichment, possible adapters are: stellar,hbaseThreatIntel` rather than a NPE. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron METRON-1490 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/963.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #963 commit a0914b2db15ec94f98d2c82173c187bfc101bc04 Author: cstella <cestella@...> Date: 2018-03-14T21:02:12Z METRON-1490: Better error message when user specifies an enrichment type that doesn't exist ---
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174478516 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/SendToKafka.java --- @@ -0,0 +1,107 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.load; + +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.ProducerRecord; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.TimerTask; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Future; +import java.util.concurrent.atomic.AtomicLong; +import java.util.function.Supplier; + +public class SendToKafka extends TimerTask { --- End diff -- Ah, so if I'm following you, making the `ThreadLocal` into `ThreadLocal<KafkaProducer<String, String>>` would solve the issue, correct? I'd prefer to not change to`SendToKafka<K, V>` because it implies that we might be sending something other than String in this implementation. We would not because the message generator sends Strings and I'd prefer to not generalize that. ---
[GitHub] metron issue #942: METRON-1461: Modify the MIN, MAX Stellar methods to take ...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/942 So, I would suggest rather than accepting a list of stats objects, `MAX` and `MIN` accept one of the following: * A StatisticsProvider object * A list of comparables Essentially, these two would both function: * `MAX(STATS_ADD(null, 1, 2, 3))` * `MAX([1, 2, 3])` ---
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174186826 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/sampler/BiasedSampler.java --- @@ -0,0 +1,95 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.sampler; + +import com.google.common.base.Splitter; +import com.google.common.collect.Iterables; + +import java.io.BufferedReader; +import java.io.File; +import java.io.FileReader; +import java.io.IOException; +import java.util.AbstractMap; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.Random; +import java.util.TreeMap; + +public class BiasedSampler implements Sampler { + TreeMap<Double, Map.Entry<Integer, Integer>> discreteDistribution; + public BiasedSampler(List<Map.Entry<Integer, Integer>> discreteDistribution, int max) { +this.discreteDistribution = createDistribution(discreteDistribution, max); + } + + public static List<Map.Entry<Integer, Integer>> readDistribution(File distrFile) throws IOException { +List<Map.Entry<Integer, Integer>> ret = new ArrayList<>(); +System.out.println("Using biased sampler with the following biases:"); +try(BufferedReader br = new BufferedReader(new FileReader(distrFile))) { + int sumLeft = 0; + int sumRight = 0; + for(String line = null;(line = br.readLine()) != null;) { +if(line.startsWith("#")) { + continue; +} +Iterable it = Splitter.on(",").split(line.trim()); +int left = Integer.parseInt(Iterables.getFirst(it, null)); +int right = Integer.parseInt(Iterables.getLast(it, null)); +System.out.println("\t" + left + "% of templates will comprise roughly " + right + "% of sample output"); +ret.add(new AbstractMap.SimpleEntry<>(left, right)); +sumLeft += left; +sumRight += right; + } + if(sumLeft > 100 || sumRight > 100 ) { +throw new IllegalStateException("Neither columns must sum to beyond 100. " + +"The first column is the % of templates. " + +"The second column is the % of the sample that % of template occupies."); + } + else if(sumLeft < 100 && sumRight < 100) { +int left = 100 - sumLeft; +int right = 100 - sumRight; +System.out.println("\t" + left + "% of templates will comprise roughly " + right + "% of sample output"); +ret.add(new AbstractMap.SimpleEntry<>(left, right)); + } + return ret; +} + } + + private static TreeMap<Double, Map.Entry<Integer, Integer>> + createDistribution(List<Map.Entry<Integer, Integer>> discreteDistribution, int max) { +TreeMap<Double, Map.Entry<Integer, Integer>> ret = new TreeMap<>(); +int from = 0; +double weight = 0.0d; +for(Map.Entry<Integer, Integer> kv : discreteDistribution) { + double pctVals = kv.getKey()/100.0; + int to = from + (int)(max*pctVals); + double pctWeight = kv.getValue()/100.0; + ret.put(weight, new AbstractMap.SimpleEntry<>(from, to)); + weight += pctWeight; + from = to; +} +return ret; + } + + @Override + public int sample(Random rng, int limit) { +double weight = rng.nextDouble(); +Map.Entry<Integer, Integer> range = discreteDistribution.floorEntry(weight).getValue(); +return rng.nextInt(range.getValue() - range.getKey()) + range.getKey(); --- End diff -- done ---
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174186723 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/sampler/BiasedSampler.java --- @@ -0,0 +1,95 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.sampler; + +import com.google.common.base.Splitter; +import com.google.common.collect.Iterables; + +import java.io.BufferedReader; +import java.io.File; +import java.io.FileReader; +import java.io.IOException; +import java.util.AbstractMap; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.Random; +import java.util.TreeMap; + +public class BiasedSampler implements Sampler { + TreeMap<Double, Map.Entry<Integer, Integer>> discreteDistribution; + public BiasedSampler(List<Map.Entry<Integer, Integer>> discreteDistribution, int max) { +this.discreteDistribution = createDistribution(discreteDistribution, max); + } + + public static List<Map.Entry<Integer, Integer>> readDistribution(File distrFile) throws IOException { +List<Map.Entry<Integer, Integer>> ret = new ArrayList<>(); +System.out.println("Using biased sampler with the following biases:"); +try(BufferedReader br = new BufferedReader(new FileReader(distrFile))) { + int sumLeft = 0; + int sumRight = 0; + for(String line = null;(line = br.readLine()) != null;) { +if(line.startsWith("#")) { + continue; +} +Iterable it = Splitter.on(",").split(line.trim()); +int left = Integer.parseInt(Iterables.getFirst(it, null)); --- End diff -- done ---
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174186700 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/monitor/writers/Writer.java --- @@ -0,0 +1,91 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.load.monitor.writers; + +import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics; +import org.apache.metron.performance.load.monitor.AbstractMonitor; +import org.apache.metron.performance.load.monitor.Results; + +import java.util.ArrayList; +import java.util.Date; +import java.util.LinkedList; +import java.util.List; +import java.util.Optional; +import java.util.function.Consumer; + +public class Writer { + + private int summaryLookback; + private List<LinkedList> summaries = new ArrayList<>(); + private List<Consumer> writers; + private List monitors; + + public Writer(List monitors, int summaryLookback, List<Consumer> writers) { +this.summaryLookback = summaryLookback; +this.writers = writers; +this.monitors = monitors; +for(AbstractMonitor m : monitors) { + this.summaries.add(new LinkedList<>()); +} + } + + public void writeAll() { +int i = 0; +Date dateOf = new Date(); +List results = new ArrayList<>(); +for(AbstractMonitor m : monitors) { + Long eps = m.get(); + if(eps != null) { +if (summaryLookback > 0) { + LinkedList summary = summaries.get(i); + addToLookback(eps == null ? Double.NaN : eps.doubleValue(), summary); + results.add(new Results(m.format(), m.name(), eps, Optional.of(getStats(summary; +} +else { + results.add(new Results(m.format(),m.name(), eps, Optional.empty())); +} + } + else { --- End diff -- done ---
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174186800 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/sampler/UnbiasedSampler.java --- @@ -0,0 +1,28 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.sampler; + +import java.util.Random; + +public class UnbiasedSampler implements Sampler { + + @Override + public int sample(Random rng, int limit) { +return rng.nextInt(limit); --- End diff -- done ---
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174186641 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/monitor/writers/Writer.java --- @@ -0,0 +1,91 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.load.monitor.writers; + +import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics; +import org.apache.metron.performance.load.monitor.AbstractMonitor; +import org.apache.metron.performance.load.monitor.Results; + +import java.util.ArrayList; +import java.util.Date; +import java.util.LinkedList; +import java.util.List; +import java.util.Optional; +import java.util.function.Consumer; + +public class Writer { + + private int summaryLookback; + private List<LinkedList> summaries = new ArrayList<>(); + private List<Consumer> writers; + private List monitors; + + public Writer(List monitors, int summaryLookback, List<Consumer> writers) { +this.summaryLookback = summaryLookback; +this.writers = writers; +this.monitors = monitors; +for(AbstractMonitor m : monitors) { + this.summaries.add(new LinkedList<>()); +} + } + + public void writeAll() { +int i = 0; +Date dateOf = new Date(); +List results = new ArrayList<>(); +for(AbstractMonitor m : monitors) { + Long eps = m.get(); + if(eps != null) { +if (summaryLookback > 0) { + LinkedList summary = summaries.get(i); + addToLookback(eps == null ? Double.NaN : eps.doubleValue(), summary); --- End diff -- done ---
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174186597 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/monitor/writers/ConsoleWriter.java --- @@ -0,0 +1,67 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.load.monitor.writers; + +import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics; +import org.apache.metron.performance.load.monitor.Results; + +import java.text.DateFormat; +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; +import java.util.function.Consumer; + +public class ConsoleWriter implements Consumer { + + private String getSummary(DescriptiveStatistics stats) { +return String.format("Mean: %d, Std Dev: %d", (int)stats.getMean(), (int)Math.sqrt(stats.getVariance())); + } + + @Override + public void accept(Writable writable) { +List parts = new ArrayList<>(); +Date date = writable.getDate(); +for(Results r : writable.getResults()) { + Long eps = r.getEps(); + if(eps != null) { +String part = String.format(r.getFormat(), eps); +if (r.getHistory().isPresent()) { + part += " (" + getSummary(r.getHistory().get()) + ")"; +} +parts.add(part); + } +} +if(date != null) { + DateFormat dateFormat = new SimpleDateFormat("/MM/dd HH:mm:ss"); + String header = dateFormat.format(date) + " - "; + String emptyHeader = ""; + for (int i = 0; i < header.length(); ++i) { --- End diff -- done ---
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174186529 --- Diff: metron-contrib/metron-performance/src/test/java/org/apache/metron/performance/load/SendToKafkaTest.java --- @@ -0,0 +1,49 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.load; + +import org.apache.kafka.clients.producer.KafkaProducer; +import org.junit.Assert; +import org.junit.Test; + +import java.util.concurrent.ExecutorService; +import java.util.concurrent.ForkJoinPool; +import java.util.concurrent.Future; +import java.util.concurrent.atomic.AtomicLong; + +public class SendToKafkaTest { + + @Test + public void testWritesCorrectNumber() throws InterruptedException { --- End diff -- done ---
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174186458 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadOptions.java --- @@ -0,0 +1,504 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.load; + +import com.google.common.base.Joiner; +import org.apache.commons.cli.*; +import org.apache.metron.common.utils.JSONUtils; +import org.apache.metron.common.utils.cli.CLIOptions; +import org.apache.metron.performance.sampler.BiasedSampler; +import org.apache.metron.stellar.common.utils.ConversionUtils; +import org.apache.metron.common.utils.cli.OptionHandler; + +import javax.annotation.Nullable; +import java.io.BufferedReader; +import java.io.File; +import java.io.FileNotFoundException; +import java.io.FileReader; +import java.io.IOException; +import java.util.ArrayList; +import java.util.EnumMap; +import java.util.List; +import java.util.Optional; + +public enum LoadOptions implements CLIOptions { + HELP(new OptionHandler() { + +@Override +public String getShortCode() { + return "h"; +} + +@Nullable +@Override +public Option apply(@Nullable String s) { + return new Option(s, "help", false, "Generate Help screen"); +} + }), + ZK(new OptionHandler() { +@Nullable +@Override +public Option apply(@Nullable String s) { + Option o = new Option(s, "zk_quorum", true, "zookeeper quorum"); + o.setArgName("QUORUM"); + o.setRequired(false); + return o; +} + +@Override +public Optional getValue(LoadOptions option, CommandLine cli) { + if(option.has(cli)) { +return Optional.ofNullable(option.get(cli)); + } + else { +return Optional.empty(); + } +} + +@Override +public String getShortCode() { + return "z"; +} + }), + CONSUMER_GROUP(new OptionHandler() { +@Nullable +@Override +public Option apply(@Nullable String s) { + Option o = new Option(s, "consumer_group", true, "Consumer Group. The default is " + LoadGenerator.CONSUMER_GROUP); + o.setArgName("GROUP_ID"); + o.setRequired(false); + return o; +} + +@Override +public Optional getValue(LoadOptions option, CommandLine cli) { + if(option.has(cli)) { +return Optional.ofNullable(option.get(cli)); + } + else { +return Optional.of(LoadGenerator.CONSUMER_GROUP); + } +} + +@Override +public String getShortCode() { + return "cg"; +} + }), + BIASED_SAMPLE(new OptionHandler() { +@Nullable +@Override +public Option apply(@Nullable String s) { + Option o = new Option(s, "sample_bias", true, "The discrete distribution to bias the sampling. " + + "This is a CSV of 2 columns. The first column is the % of the templates " + + "and the 2nd column is the probability (0-100) that it's chosen. For instance:\n" + + " 20,80\n" + + " 80,20\n" + + "implies that 20% of the templates will comprise 80% of the output and the remaining 80% of the templates will comprise 20% of the output."); + o.setArgName("BIAS_FILE"); + o.setRequired(false); + return o; +} + +@Override +public Optional getValue(LoadOptions option, CommandLine cli) { + if(!option.has(cli)) { +return Optional.empty(); + } +
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174186359 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadGenerator.java --- @@ -0,0 +1,165 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.load; + + +import com.google.common.base.Joiner; +import org.apache.commons.cli.CommandLine; +import org.apache.commons.cli.PosixParser; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.metron.common.utils.KafkaUtils; +import org.apache.metron.performance.load.monitor.AbstractMonitor; +import org.apache.metron.performance.load.monitor.EPSGeneratedMonitor; +import org.apache.metron.performance.load.monitor.EPSThroughputWrittenMonitor; +import org.apache.metron.performance.load.monitor.MonitorTask; +import org.apache.metron.performance.load.monitor.writers.CSVWriter; +import org.apache.metron.performance.load.monitor.writers.ConsoleWriter; +import org.apache.metron.performance.load.monitor.writers.Writable; +import org.apache.metron.performance.load.monitor.writers.Writer; +import org.apache.metron.performance.sampler.BiasedSampler; +import org.apache.metron.performance.sampler.Sampler; +import org.apache.metron.performance.sampler.UnbiasedSampler; + +import java.io.File; +import java.util.ArrayList; +import java.util.EnumMap; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Timer; +import java.util.TimerTask; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.atomic.AtomicLong; +import java.util.function.Consumer; + +public class LoadGenerator +{ + public static String CONSUMER_GROUP = "load.group"; + public static long SEND_PERIOD_MS = 100; + public static long MONITOR_PERIOD_MS = 1000*10; + private static ExecutorService pool; + private static ThreadLocal kafkaProducer; + public static AtomicLong numSent = new AtomicLong(0); + + public static void main( String[] args ) throws Exception { +CommandLine cli = LoadOptions.parse(new PosixParser(), args); +EnumMap<LoadOptions, Optional> evaluatedArgs = LoadOptions.createConfig(cli); +Map<String, Object> kafkaConfig = new HashMap<>(); +kafkaConfig.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); +kafkaConfig.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); +kafkaConfig.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); +kafkaConfig.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); +if(LoadOptions.ZK.has(cli)) { + String zkQuorum = (String) evaluatedArgs.get(LoadOptions.ZK).get(); + kafkaConfig.put("bootstrap.servers", Joiner.on(",").join(KafkaUtils.INSTANCE.getBrokersFromZookeeper(zkQuorum))); --- End diff -- done ---
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174186403 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadGenerator.java --- @@ -0,0 +1,165 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.load; + + +import com.google.common.base.Joiner; +import org.apache.commons.cli.CommandLine; +import org.apache.commons.cli.PosixParser; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.metron.common.utils.KafkaUtils; +import org.apache.metron.performance.load.monitor.AbstractMonitor; +import org.apache.metron.performance.load.monitor.EPSGeneratedMonitor; +import org.apache.metron.performance.load.monitor.EPSThroughputWrittenMonitor; +import org.apache.metron.performance.load.monitor.MonitorTask; +import org.apache.metron.performance.load.monitor.writers.CSVWriter; +import org.apache.metron.performance.load.monitor.writers.ConsoleWriter; +import org.apache.metron.performance.load.monitor.writers.Writable; +import org.apache.metron.performance.load.monitor.writers.Writer; +import org.apache.metron.performance.sampler.BiasedSampler; +import org.apache.metron.performance.sampler.Sampler; +import org.apache.metron.performance.sampler.UnbiasedSampler; + +import java.io.File; +import java.util.ArrayList; +import java.util.EnumMap; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Timer; +import java.util.TimerTask; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.atomic.AtomicLong; +import java.util.function.Consumer; + +public class LoadGenerator +{ + public static String CONSUMER_GROUP = "load.group"; + public static long SEND_PERIOD_MS = 100; + public static long MONITOR_PERIOD_MS = 1000*10; + private static ExecutorService pool; + private static ThreadLocal kafkaProducer; + public static AtomicLong numSent = new AtomicLong(0); + + public static void main( String[] args ) throws Exception { +CommandLine cli = LoadOptions.parse(new PosixParser(), args); +EnumMap<LoadOptions, Optional> evaluatedArgs = LoadOptions.createConfig(cli); +Map<String, Object> kafkaConfig = new HashMap<>(); +kafkaConfig.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); +kafkaConfig.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); +kafkaConfig.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); +kafkaConfig.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); +if(LoadOptions.ZK.has(cli)) { + String zkQuorum = (String) evaluatedArgs.get(LoadOptions.ZK).get(); + kafkaConfig.put("bootstrap.servers", Joiner.on(",").join(KafkaUtils.INSTANCE.getBrokersFromZookeeper(zkQuorum))); +} +String groupId = evaluatedArgs.get(LoadOptions.CONSUMER_GROUP).get().toString(); +System.out.println("Consumer Group: " + groupId); +kafkaConfig.put("group.id", groupId); --- End diff -- done ---
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174186422 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadOptions.java --- @@ -0,0 +1,504 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.load; + +import com.google.common.base.Joiner; +import org.apache.commons.cli.*; +import org.apache.metron.common.utils.JSONUtils; +import org.apache.metron.common.utils.cli.CLIOptions; +import org.apache.metron.performance.sampler.BiasedSampler; +import org.apache.metron.stellar.common.utils.ConversionUtils; +import org.apache.metron.common.utils.cli.OptionHandler; + +import javax.annotation.Nullable; +import java.io.BufferedReader; +import java.io.File; +import java.io.FileNotFoundException; +import java.io.FileReader; +import java.io.IOException; +import java.util.ArrayList; +import java.util.EnumMap; +import java.util.List; +import java.util.Optional; + +public enum LoadOptions implements CLIOptions { + HELP(new OptionHandler() { + +@Override +public String getShortCode() { + return "h"; +} + +@Nullable +@Override +public Option apply(@Nullable String s) { + return new Option(s, "help", false, "Generate Help screen"); +} + }), + ZK(new OptionHandler() { +@Nullable +@Override +public Option apply(@Nullable String s) { + Option o = new Option(s, "zk_quorum", true, "zookeeper quorum"); + o.setArgName("QUORUM"); + o.setRequired(false); + return o; +} + +@Override +public Optional getValue(LoadOptions option, CommandLine cli) { + if(option.has(cli)) { +return Optional.ofNullable(option.get(cli)); + } + else { +return Optional.empty(); + } +} + +@Override +public String getShortCode() { + return "z"; +} + }), + CONSUMER_GROUP(new OptionHandler() { +@Nullable +@Override +public Option apply(@Nullable String s) { + Option o = new Option(s, "consumer_group", true, "Consumer Group. The default is " + LoadGenerator.CONSUMER_GROUP); + o.setArgName("GROUP_ID"); + o.setRequired(false); + return o; +} + +@Override +public Optional getValue(LoadOptions option, CommandLine cli) { + if(option.has(cli)) { +return Optional.ofNullable(option.get(cli)); + } + else { +return Optional.of(LoadGenerator.CONSUMER_GROUP); + } +} + +@Override +public String getShortCode() { + return "cg"; +} + }), + BIASED_SAMPLE(new OptionHandler() { +@Nullable +@Override +public Option apply(@Nullable String s) { + Option o = new Option(s, "sample_bias", true, "The discrete distribution to bias the sampling. " + + "This is a CSV of 2 columns. The first column is the % of the templates " + + "and the 2nd column is the probability (0-100) that it's chosen. For instance:\n" + + " 20,80\n" + + " 80,20\n" + + "implies that 20% of the templates will comprise 80% of the output and the remaining 80% of the templates will comprise 20% of the output."); + o.setArgName("BIAS_FILE"); + o.setRequired(false); + return o; +} + +@Override +public Optional getValue(LoadOptions option, CommandLine cli) { + if(!option.has(cli)) { +return Optional.empty(); + } +
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174186490 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/MessageGenerator.java --- @@ -0,0 +1,48 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.load; + +import org.apache.metron.performance.sampler.Sampler; + +import java.util.List; +import java.util.Random; +import java.util.concurrent.atomic.AtomicLong; +import java.util.function.Supplier; + +public class MessageGenerator implements Supplier { + private static ThreadLocal rng = ThreadLocal.withInitial(() -> new Random()); + private static AtomicLong guidOffset = new AtomicLong(0); + private static String guidPrefix = "6141faf6-a8ba-4044-ab80-"; --- End diff -- done ---
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174186316 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadGenerator.java --- @@ -0,0 +1,165 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.load; + + +import com.google.common.base.Joiner; +import org.apache.commons.cli.CommandLine; +import org.apache.commons.cli.PosixParser; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.metron.common.utils.KafkaUtils; +import org.apache.metron.performance.load.monitor.AbstractMonitor; +import org.apache.metron.performance.load.monitor.EPSGeneratedMonitor; +import org.apache.metron.performance.load.monitor.EPSThroughputWrittenMonitor; +import org.apache.metron.performance.load.monitor.MonitorTask; +import org.apache.metron.performance.load.monitor.writers.CSVWriter; +import org.apache.metron.performance.load.monitor.writers.ConsoleWriter; +import org.apache.metron.performance.load.monitor.writers.Writable; +import org.apache.metron.performance.load.monitor.writers.Writer; +import org.apache.metron.performance.sampler.BiasedSampler; +import org.apache.metron.performance.sampler.Sampler; +import org.apache.metron.performance.sampler.UnbiasedSampler; + +import java.io.File; +import java.util.ArrayList; +import java.util.EnumMap; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Timer; +import java.util.TimerTask; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.atomic.AtomicLong; +import java.util.function.Consumer; + +public class LoadGenerator +{ + public static String CONSUMER_GROUP = "load.group"; --- End diff -- done ---
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174186275 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadGenerator.java --- @@ -0,0 +1,165 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.load; + + +import com.google.common.base.Joiner; +import org.apache.commons.cli.CommandLine; +import org.apache.commons.cli.PosixParser; +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.metron.common.utils.KafkaUtils; +import org.apache.metron.performance.load.monitor.AbstractMonitor; +import org.apache.metron.performance.load.monitor.EPSGeneratedMonitor; +import org.apache.metron.performance.load.monitor.EPSThroughputWrittenMonitor; +import org.apache.metron.performance.load.monitor.MonitorTask; +import org.apache.metron.performance.load.monitor.writers.CSVWriter; +import org.apache.metron.performance.load.monitor.writers.ConsoleWriter; +import org.apache.metron.performance.load.monitor.writers.Writable; +import org.apache.metron.performance.load.monitor.writers.Writer; +import org.apache.metron.performance.sampler.BiasedSampler; +import org.apache.metron.performance.sampler.Sampler; +import org.apache.metron.performance.sampler.UnbiasedSampler; + +import java.io.File; +import java.util.ArrayList; +import java.util.EnumMap; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Timer; +import java.util.TimerTask; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.atomic.AtomicLong; +import java.util.function.Consumer; + +public class LoadGenerator +{ + public static String CONSUMER_GROUP = "load.group"; + public static long SEND_PERIOD_MS = 100; + public static long MONITOR_PERIOD_MS = 1000*10; + private static ExecutorService pool; + private static ThreadLocal kafkaProducer; + public static AtomicLong numSent = new AtomicLong(0); + + public static void main( String[] args ) throws Exception { +CommandLine cli = LoadOptions.parse(new PosixParser(), args); +EnumMap<LoadOptions, Optional> evaluatedArgs = LoadOptions.createConfig(cli); +Map<String, Object> kafkaConfig = new HashMap<>(); +kafkaConfig.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); --- End diff -- done ---
[GitHub] metron issue #958: METRON-1483: Create a tool to monitor performance of the ...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/958 @justinleet thanks for the review. I reacted to your comments either by fixing them or suggesting why I prefer what is there. I will add a new set of tests in the next commit. ---
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r174181759 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/SendToKafka.java --- @@ -0,0 +1,107 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.load; + +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.ProducerRecord; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.Collections; +import java.util.TimerTask; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Future; +import java.util.concurrent.atomic.AtomicLong; +import java.util.function.Supplier; + +public class SendToKafka extends TimerTask { --- End diff -- well, I mean, I suppose it could be, but I didn't see value in it since all of our inputs currently are strings and the message generator exclusively generates strings, not other messages. It'd be a more serious refactoring to genericize this to make the message generator pluggable. Also, for utility code, I think I'd rather avoid prematurely generalizing it. Let me know what you think. ---
[GitHub] metron pull request #958: METRON-1483: Create a tool to monitor performance ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/958#discussion_r173942355 --- Diff: metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/monitor/AbstractMonitor.java --- @@ -0,0 +1,49 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.performance.load.monitor; + +import java.util.Optional; +import java.util.function.Supplier; + +public abstract class AbstractMonitor implements Supplier, MonitorNaming { + private static final double EPSILON = 1e-6; + protected Optional kafkaTopic; + protected long timestampPrevious = 0; + public AbstractMonitor(Optional kafkaTopic) { +this.kafkaTopic = kafkaTopic; + } + + protected abstract Long monitor(double deltaTs); + + @Override + public Long get() { +long timeStarted = System.currentTimeMillis(); --- End diff -- To my knowledge, `currentTimeMillis()` is not affected by DST as it is the milliseconds since unix epoch UTC. Cursory googling seems to hold up this assertion. Did I miss something? ---
[GitHub] metron pull request #961: METRON-1487 Define Performance Benchmarks for Enri...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/961#discussion_r173928377 --- Diff: metron-platform/metron-enrichment/Performance.md --- @@ -0,0 +1,522 @@ + + +# Enrichment Performance + +This guide defines a set of benchmarks used to measure the performance of the Enrichment topology. The guide also provides detailed steps on how to execute those benchmarks along with advice for tuning the Enrichment topology. + +* [Benchmarks](#benchmarks) +* [Benchmark Execution](#benchmark-execution) +* [Performance Tuning](#performance-tuning) +* [Benchmark Results](#benchmark-results) + +## Benchmarks + +* [Geo IP Enrichment](#geo-ip-enrichment) +* [HBase Enrichment](#hbase-enrichment) +* [Stellar Enrichment](#stellar-enrichment) + +### Geo IP Enrichment + +This benchmark measures the performance of executing a Geo IP enrichment. Given a valid IP address the enrichment will append detailed location information for that IP. The location information is sourced from an external Geo IP data source like [Maxmind](https://github.com/maxmind/GeoIP2-java). + + Configuration + +Adding the following Stellar expression to the Enrichment topology configuration will define a Geo IP enrichment. +``` +geo := GEO_GET(ip_dst_addr) +``` + +After the enrichment process completes, the telemetry message will contain a set of fields with location information for the given IP address. +``` +{ + "ip_dst_addr":"151.101.129.140", + ... + "geo.city":"San Francisco", + "geo.country":"US", + "geo.dmaCode":"807", + "geo.latitude":"37.7697", + "geo.location_point":"37.7697,-122.3933", + "geo.locID":"5391959", + "geo.longitude":"-122.3933", + "geo.postalCode":"94107", + } +``` + +### HBase Enrichment + +This benchmark measures the performance of executing an enrichment that retrieves data from an external HBase table. This type of enrichment is useful for enriching telemetry from an Asset Database or other source of relatively static data. + + Configuration + +Adding the following Stellar expression to the Enrichment topology configuration will define an Hbase enrichment. This looks up the 'ip_dst_addr' within an HBase table 'top-1m' and returns a hostname. +``` +top1m := ENRICHMENT_GET('top-1m', ip_dst_addr, 'top-1m', 't') +``` + +After the telemetry has been enriched, it will contain the host and IP elements that were retrieved from the HBase table. +``` +{ + "ip_dst_addr":"151.101.2.166", + ... + "top1m.host":"earther.com", + "top1m.ip":"151.101.2.166" +} +``` + +### Stellar Enrichment + +This benchmark measures the performance of executing a basic Stellar expression. In this benchmark, the enrichment is purely a computational task that has no dependence on an external system like a database. + + Configuration + +Adding the following Stellar expression to the Enrichment topology configuration will define a basic Stellar enrichment. The following returns true if the IP is in the given subnet and false otherwise. +``` +local := IN_SUBNET(ip_dst_addr, '192.168.0.0/24') +``` + +After the telemetry has been enriched, it will contain a field with a boolean value indicating whether the IP was within the given subnet. +``` +{ + "ip_dst_addr":"151.101.2.166", + ... + "local":false +} +``` + +## Benchmark Execution + +* [Prepare Enrichment Data](#prepare-enrichment-data) +* [Load HBase with Enrichment Data](#load-hbase-with-enrichment-data) +* [Configure the Enrichments](#configure-the-enrichments) +* [Create Input Telemetry](#create-input-telemetry) +* [Cluster Setup](#cluster-setup) +* [Monitoring](#monitoring) + +### Prepare Enrichment Data + +The Alexa Top 1 Million was used as an data source for these benchmarks. + +1. Download the [Alexa Top 1 Million](http://s3.amazonaws.com/alexa-static/top-1m.csv.zip). + +2. For each hostname, query DNS to retrieve an associated IP address. + + A script like the following can be used for this. There is no need to do this for all 1 million entries in the data set. Doing this for around 10,000 records is sufficient. + + ```python + import dns.resolver + import csv + + resolver = dns.resolver.Resolver() + resolver.nameservers = ['8.8.8.8',
[GitHub] metron pull request #962: METRON-1488: user_settings hbase table does not ha...
GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/962 METRON-1488: user_settings hbase table does not have acls set up for kerberos ## Contributor Comments Currently some REST calls will fail because we do not set ACL's on the new user_settings table, which is new. To exercise this, kerberize full-dev (or whatever cluster you please) and go to swagger and try to do a search via `api/v1/search/search` with the following arg: ``` { "indices": [], "facetFields":[], "query": "*", "from": 0, "size": 20 } ``` This should succeed now. Before we got the following exception: ``` { "responseCode": 500, "message": "org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'metron' (table=user_settings, action=READ)\n\tat org.apache.hadoop.hbase.security.access.AccessController.internalPreRead(AccessController.java:1616)\n\tat org.apache.hadoop.hbase.security.access.AccessController.preGetOp(AccessController.java:1624)\n\tat org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$26.call(RegionCoprocessorHost.java:816)\n\tat org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1660)\n\tat org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1734)\n\tat org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1692)\n\tat org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:812)\n\tat org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6917)\n\tat org.apa che.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6905)\n\tat org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2026)\n\tat org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32381)\n\tat org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)\n\tat org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)\n", "fullMessage": "RemoteWithExtrasException: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'metron' (table=user_settings, action=READ)\n\tat org.apache.hadoop.hbase.security.access.AccessController.internalPreRead(AccessController.java:1616)\n\tat org.apache.hadoop.hbase.security.access.AccessController.preGetOp(AccessController.java:1624)\n\tat org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$26.call(RegionCoprocessorHost.java:816)\n\tat org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1660)\n\tat org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1734)\n\tat org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1692)\n\tat org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:812)\n\tat org.apache.hadoop.hbase.regionserver.HRegion.get(H Region.java:6917)\n\tat org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6905)\n\tat org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2026)\n\tat org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32381)\n\tat org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)\n\tat org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)\n\tat org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)\n" } ``` ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR t
[GitHub] metron issue #958: METRON-1483: Create a tool to monitor performance of the ...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/958 # Testing Plan: We presume * `ZOOKEEPER` is an environment variable set to the zk quorum (e.g. `node1:2181`) * `BROKER` is an environment variable set to the broker (e.g. `node1:6667`) ## Unbiased Generation In order to test this, we will: 1. create a new topic ``` # Create dummy_12_unbiased with 12 partitions /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER --create --topic dummy_12_unbiased --partitions 12 --replication-factor 1 ``` 2. Create some dummy data to send into kafka ``` # Generate some templates of dummy data that look like CSV in ~/dummy.templates # With a number, a string, a GUID and the timestamp for i in $(seq 1 100);do echo "$i,foo,\$METRON_GUID,\$METRON_TS";done > ~/dummy.templates ``` 3. Write into that topic from a set of templates at a specified rate ``` # Write out 1000 messages per second to dummy_12_unbiased # Monitor dummy_12_unbiased # Each message is drawn from the set of templates in step 2. # Stop after 60 seconds and report how much you've written $METRON_HOME/bin/load_tool.sh -p 5 -ot dummy_12_unbiased -mt dummy_12_unbiased -t ~/dummy.template -eps 1000 -z $ZOOKEEPER -tl 6 ``` * It should indicate that it generated something like 60k messages 4. Measure if the messages generated roughly match the requested rate. ``` # You will have to ctrl-c this in a few seconds /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --bootstrap-server node1:6667 --topic dummy_12_topic --from-beginning | wc -l ``` * It should output something like `Processed a total of 60400 messages`, which would be about 1000 messages per second. ---
[GitHub] metron pull request #949: METRON-1471: Migrate shuffle connections to local ...
Github user cestella closed the pull request at: https://github.com/apache/metron/pull/949 ---
[GitHub] metron pull request #949: METRON-1471: Migrate shuffle connections to local ...
GitHub user cestella reopened a pull request: https://github.com/apache/metron/pull/949 METRON-1471: Migrate shuffle connections to local or shuffle ## Contributor Comments Currently, we use shuffle groupings when we do not want to group by field. We should, instead, use local or shuffle groupings. Manual tests should include ensuring data continues to flow as before. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron local_or_shuffle_indexing Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/949.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #949 commit 0ab26fa172a90eaf8eb441c8c085308649efdfb9 Author: cstella <cestella@...> Date: 2018-03-06T16:18:57Z Updating the rest of the topologies to prefer local or shuffle over shuffle commit 82a7c823ab27441ddda5c043dee3cef8aa8f03be Author: cstella <cestella@...> Date: 2018-03-08T14:25:19Z Merge branch 'master' into local_or_shuffle_indexing commit 2d3a7f68dc7570ca1a44b9b6a4b2880292e18824 Author: cstella <cestella@...> Date: 2018-03-08T14:26:48Z Fixed profiler ---
[GitHub] metron pull request #958: METRON-1483: In performance evaluation, generating...
GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/958 METRON-1483: In performance evaluation, generating synthetic load and monitoring the write throughput of our kafka-to-kafka topologies has required a lot of custom scripting. We should have a tool that could do the following: Take a file representing a message template and generate synthetic load at a given events per second Monitor the kafka offsets of a topic and report throughput numbers in events per second ## Contributor Comments In performance evaluation, generating synthetic load and monitoring the write throughput of our kafka-to-kafka topologies has required a lot of custom scripting. We should have a tool that could do the following: * Take a file representing a message template and generate synthetic load at a given events per second * Monitor the kafka offsets of a topic and report throughput numbers in events per second Currently pending: * Test plan pending * Documentation ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron perf_tool Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/958.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #958 commit 8ba8c57576ed318d1ff3f69cda1dc9020d62352a Author: cstella <cestella@...> Date: 2018-03-08T00:03:33Z Build load testing tool commit 638086f2f30cfe543b624721c1fb467062571eab Author: cstella <cestella@...> Date: 2018-03-08T15:31:55Z refactoring monitoring to be more sensible. commit 55be01de033d0ea6b489930cb97fefcca80efdb5 Author: cstella <cestella@...> Date: 2018-03-08T16:20:52Z Added summarization ---
[GitHub] metron issue #945: METRON-1464: Convert schemas to be compatible with Solr 5...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/945 Ok, I'm cool with it. +1 by inspection; great work. ---
[GitHub] metron issue #949: METRON-1471: Migrate shuffle connections to local or shuf...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/949 @merrimanr Ah, I had forgotten to merge in master yesterday afternoon. There should be no more shuffles in any of the topologies. ---
[GitHub] metron pull request #947: METRON-1467: Replace guava caches in places where ...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/947#discussion_r172870992 --- Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java --- @@ -89,29 +91,25 @@ public void prepare(Map map, TopologyContext topologyContext, OutputCollector ou if (this.maxTimeRetain == null) { throw new IllegalStateException("maxTimeRetain must be specified"); } -loader = new CacheLoader<String, Map<String, Tuple>>() { - @Override - public Map<String, Tuple> load(String key) throws Exception { -return new HashMap<>(); - } -}; -cache = CacheBuilder.newBuilder().maximumSize(maxCacheSize) -.expireAfterWrite(maxTimeRetain, TimeUnit.MINUTES).removalListener(new JoinRemoveListener()) -.build(loader); +loader = s -> new HashMap<>(); +cache = Caffeine.newBuilder().maximumSize(maxCacheSize) + .expireAfterWrite(maxTimeRetain, TimeUnit.MINUTES) + .removalListener(new JoinRemoveListener()) --- End diff -- So, I believe this was intentionally done before this PR (I migrated this to the new caching strategy) and the idea is that if a removal is happening from the join cache under specific circumstances, we want to know about it because a message could be being dropped because the cache is being overwhelmed. @merrimanr Can you chime in here on the rationale? ---
[GitHub] metron issue #945: METRON-1464: Convert schemas to be compatible with Solr 5...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/945 @merrimanr you ran this up in Solr 5.5 as well as 6.6, right? If so, then I'm content with the change and give a +1 pending (other than holding for an answer to Simon's question, which I was wondering as well). ---
[GitHub] metron pull request #952: Metron-1480 Add yarn as default build tool for the...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/952#discussion_r172858774 --- Diff: metron-interface/metron-alerts/package-lock.json --- @@ -1,6427 +0,0 @@ -{ --- End diff -- I suspect the build failure is due to a lack of a license on this file. Since JSON cannot take comments, you'll need to adjust the top level pom to add an exclusion for `**/package-lock.json` [here](https://github.com/apache/metron/blob/master/pom.xml#L304) Just make a quick comment above it and note that this is yarn collateral. ---
[GitHub] metron pull request #940: METRON-1460: Create a complementary non-split-join...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/940#discussion_r172711543 --- Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java --- @@ -0,0 +1,281 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.enrichment.parallel; + +import com.github.benmanes.caffeine.cache.stats.CacheStats; +import org.apache.metron.common.Constants; +import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig; +import org.apache.metron.common.configuration.enrichment.handler.ConfigHandler; +import org.apache.metron.common.performance.PerformanceLogger; +import org.apache.metron.common.utils.MessageUtils; +import org.apache.metron.enrichment.bolt.CacheKey; +import org.apache.metron.enrichment.interfaces.EnrichmentAdapter; +import org.apache.metron.enrichment.utils.EnrichmentUtils; +import org.json.simple.JSONObject; + +import java.util.AbstractMap; +import java.util.ArrayList; +import java.util.Collections; +import java.util.EnumMap; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; +import java.util.function.BinaryOperator; +import java.util.function.Supplier; + +/** + * This is an independent component which will accept a message and a set of enrichment adapters as well as a config which defines + * how those enrichments should be performed and fully enrich the message. The result will be the enriched message + * unified together and a list of errors which happened. + */ +public class ParallelEnricher { + + private Map<String, EnrichmentAdapter> enrichmentsByType = new HashMap<>(); + private EnumMap<EnrichmentStrategies, CacheStats> cacheStats = new EnumMap<>(EnrichmentStrategies.class); + + /** + * The result of an enrichment. + */ + public static class EnrichmentResult { +private JSONObject result; +private List<Map.Entry<Object, Throwable>> enrichmentErrors; + +public EnrichmentResult(JSONObject result, List<Map.Entry<Object, Throwable>> enrichmentErrors) { + this.result = result; + this.enrichmentErrors = enrichmentErrors; +} + +/** + * The unified fully enriched result. + * @return + */ +public JSONObject getResult() { + return result; +} + +/** + * The errors that happened in the course of enriching. + * @return + */ +public List<Map.Entry<Object, Throwable>> getEnrichmentErrors() { + return enrichmentErrors; +} + } + + private ConcurrencyContext concurrencyContext; + + /** + * Construct a parallel enricher with a set of enrichment adapters associated with their enrichment types. + * @param enrichmentsByType + */ + public ParallelEnricher( Map<String, EnrichmentAdapter> enrichmentsByType + , ConcurrencyContext concurrencyContext + , boolean logStats + ) + { +this.enrichmentsByType = enrichmentsByType; +this.concurrencyContext = concurrencyContext; +if(logStats) { + for(EnrichmentStrategies s : EnrichmentStrategies.values()) { +cacheStats.put(s, null); + } +} + } + + /** + * Fully enriches a message. Each enrichment is done in parallel via a threadpool. + * Each enrichment is fronted with a LRU cache. + * + * @param message the message to enrich + * @param strategy The enrichment strategy to use (e.g. enrichment or threat intel) + * @param config The sensor enri
[GitHub] metron pull request #940: METRON-1460: Create a complementary non-split-join...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/940#discussion_r172656809 --- Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java --- @@ -0,0 +1,281 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.enrichment.parallel; + +import com.github.benmanes.caffeine.cache.stats.CacheStats; +import org.apache.metron.common.Constants; +import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig; +import org.apache.metron.common.configuration.enrichment.handler.ConfigHandler; +import org.apache.metron.common.performance.PerformanceLogger; +import org.apache.metron.common.utils.MessageUtils; +import org.apache.metron.enrichment.bolt.CacheKey; +import org.apache.metron.enrichment.interfaces.EnrichmentAdapter; +import org.apache.metron.enrichment.utils.EnrichmentUtils; +import org.json.simple.JSONObject; + +import java.util.AbstractMap; +import java.util.ArrayList; +import java.util.Collections; +import java.util.EnumMap; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; +import java.util.function.BinaryOperator; +import java.util.function.Supplier; + +/** + * This is an independent component which will accept a message and a set of enrichment adapters as well as a config which defines + * how those enrichments should be performed and fully enrich the message. The result will be the enriched message + * unified together and a list of errors which happened. + */ +public class ParallelEnricher { + + private Map<String, EnrichmentAdapter> enrichmentsByType = new HashMap<>(); + private EnumMap<EnrichmentStrategies, CacheStats> cacheStats = new EnumMap<>(EnrichmentStrategies.class); + + /** + * The result of an enrichment. + */ + public static class EnrichmentResult { +private JSONObject result; +private List<Map.Entry<Object, Throwable>> enrichmentErrors; + +public EnrichmentResult(JSONObject result, List<Map.Entry<Object, Throwable>> enrichmentErrors) { + this.result = result; + this.enrichmentErrors = enrichmentErrors; +} + +/** + * The unified fully enriched result. + * @return + */ +public JSONObject getResult() { + return result; +} + +/** + * The errors that happened in the course of enriching. + * @return + */ +public List<Map.Entry<Object, Throwable>> getEnrichmentErrors() { + return enrichmentErrors; +} + } + + private ConcurrencyContext concurrencyContext; + + /** + * Construct a parallel enricher with a set of enrichment adapters associated with their enrichment types. + * @param enrichmentsByType + */ + public ParallelEnricher( Map<String, EnrichmentAdapter> enrichmentsByType + , ConcurrencyContext concurrencyContext + , boolean logStats + ) + { +this.enrichmentsByType = enrichmentsByType; +this.concurrencyContext = concurrencyContext; +if(logStats) { + for(EnrichmentStrategies s : EnrichmentStrategies.values()) { +cacheStats.put(s, null); + } +} + } + + /** + * Fully enriches a message. Each enrichment is done in parallel via a threadpool. + * Each enrichment is fronted with a LRU cache. + * + * @param message the message to enrich + * @param strategy The enrichment strategy to use (e.g. enrichment or threat intel) + * @param config The sensor enri
[GitHub] metron pull request #949: METRON-1471: Migrate shuffle connections to local ...
GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/949 METRON-1471: Migrate shuffle connections to local or shuffle ## Contributor Comments Currently, we use shuffle groupings when we do not want to group by field. We should, instead, use local or shuffle groupings. Manual tests should include ensuring data continues to flow as before. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron local_or_shuffle_indexing Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/949.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #949 commit 0ab26fa172a90eaf8eb441c8c085308649efdfb9 Author: cstella <cestella@...> Date: 2018-03-06T16:18:57Z Updating the rest of the topologies to prefer local or shuffle over shuffle ---
[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/940 Ok, README is updated with the new topology diagram. Let me know if there's anything else. ---
[GitHub] metron pull request #940: METRON-1460: Create a complementary non-split-join...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/940#discussion_r172383791 --- Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/Strategy.java --- @@ -0,0 +1,47 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.enrichment.parallel; + +import org.apache.metron.common.Constants; +import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig; +import org.apache.metron.common.configuration.enrichment.handler.ConfigHandler; +import org.json.simple.JSONObject; +import org.slf4j.Logger; + +import java.util.Map; + +/** + * Enrichment strategy. This interface provides a mechanism to interface with the enrichment config and any + * post processing steps that are needed to be done after-the-fact. + * + * The reasoning behind this is that the key difference between enrichments and threat intel is that they pull + * their configurations from different parts of the SensorEnrichmentConfig object and as a post-join step, they differ + * slightly. + * + */ +public interface Strategy { + Constants.ErrorType getErrorType(); --- End diff -- done ---
[GitHub] metron pull request #940: METRON-1460: Create a complementary non-split-join...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/940#discussion_r172383810 --- Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java --- @@ -0,0 +1,415 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.enrichment.bolt; + +import org.apache.metron.common.Constants; +import org.apache.metron.common.bolt.ConfiguredEnrichmentBolt; +import org.apache.metron.common.configuration.ConfigurationType; +import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig; +import org.apache.metron.common.error.MetronError; +import org.apache.metron.common.performance.PerformanceLogger; +import org.apache.metron.common.utils.ErrorUtils; +import org.apache.metron.common.utils.MessageUtils; +import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase; +import org.apache.metron.enrichment.configuration.Enrichment; +import org.apache.metron.enrichment.interfaces.EnrichmentAdapter; +import org.apache.metron.enrichment.parallel.EnrichmentContext; +import org.apache.metron.enrichment.parallel.EnrichmentStrategies; +import org.apache.metron.enrichment.parallel.ParallelEnricher; +import org.apache.metron.enrichment.parallel.WorkerPoolStrategy; +import org.apache.metron.stellar.dsl.Context; +import org.apache.metron.stellar.dsl.StellarFunction; +import org.apache.metron.stellar.dsl.StellarFunctions; +import org.apache.storm.task.OutputCollector; +import org.apache.storm.task.TopologyContext; +import org.apache.storm.topology.OutputFieldsDeclarer; +import org.apache.storm.tuple.Fields; +import org.apache.storm.tuple.Tuple; +import org.apache.storm.tuple.Values; +import org.json.simple.JSONObject; +import org.json.simple.parser.JSONParser; +import org.json.simple.parser.ParseException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.UnsupportedEncodingException; +import java.lang.invoke.MethodHandles; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.UUID; + +/** + * This bolt is a unified enrichment/threat intel bolt. In contrast to the split/enrich/join + * bolts above, this handles the entire enrichment lifecycle in one bolt using a threadpool to + * enrich in parallel. + * + * From an architectural perspective, this is a divergence from the polymorphism based strategy we have + * used in the split/join bolts. Rather, this bolt is provided a strategy to use, either enrichment or threat intel, + * through composition. This allows us to move most of the implementation into components independent + * from Storm. This will greater facilitate reuse. + */ +public class UnifiedEnrichmentBolt extends ConfiguredEnrichmentBolt { + + public static class Perf {} // used for performance logging + private PerformanceLogger perfLog; // not static bc multiple bolts may exist in same worker + + protected static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass()); + + public static final String STELLAR_CONTEXT_CONF = "stellarContext"; + + /** + * The number of threads in the threadpool. One threadpool is created per process. + * This is a topology-level configuration + */ + public static final String THREADPOOL_NUM_THREADS_TOPOLOGY_CONF = "metron.threadpool.size"; + /** + * The type of threadpool to create. This is a topology-level configuration. + */ + public static final String THREADPOOL_TYPE_TOPOLOGY_CONF = "metron.threadpool.type"; + + /** + * The enricher implementation to use. This will do the parallel enrichment via a thread pool. + */ + protected ParallelEnricher enricher; + + /** + * The strategy to use for this enrichment bolt. Practically speak
[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/940 @nickwallen Ok, I refactored the abstraction to separate some concerns, name things a bit, and collapse some of the more onerous abstractions. Also updated javadocs. Can you give it another look and see what you think? We probably should also give it another smoketest in the lab to make sure I didn't do something dumb. ---
[GitHub] metron pull request #940: METRON-1460: Create a complementary non-split-join...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/940#discussion_r172383754 --- Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/EnrichmentStrategies.java --- @@ -0,0 +1,79 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.enrichment.parallel; + +import org.apache.metron.common.Constants; +import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig; +import org.apache.metron.common.configuration.enrichment.handler.ConfigHandler; +import org.apache.metron.enrichment.bolt.CacheKey; +import org.json.simple.JSONObject; +import org.slf4j.Logger; + +import java.util.Map; +import java.util.concurrent.Executor; + +public enum EnrichmentStrategies implements Strategy { --- End diff -- I decided that this is too onerous of an abstraction and rethought it a bit. Give it another look, please. ---
[GitHub] metron pull request #940: METRON-1460: Create a complementary non-split-join...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/940#discussion_r172377136 --- Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/EnrichmentStrategies.java --- @@ -0,0 +1,79 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.enrichment.parallel; + +import org.apache.metron.common.Constants; +import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig; +import org.apache.metron.common.configuration.enrichment.handler.ConfigHandler; +import org.apache.metron.enrichment.bolt.CacheKey; +import org.json.simple.JSONObject; +import org.slf4j.Logger; + +import java.util.Map; +import java.util.concurrent.Executor; + +public enum EnrichmentStrategies implements Strategy { --- End diff -- I might be answering a question that you're not asking, but this bit of awkwardness arises because we have merged the concepts of threat intel and enrichment, which differ really only in post-processing. The approach presented here, in contrast to the inheritance-based approach in the bolts, allows for an abstraction through composition whereby we localize all the interactions with the sensor enrichment config in a strategy rather than bind the abstraction to Storm, our distributed processing engine. That is the rationale behind this approach at least. ---
[GitHub] metron pull request #940: METRON-1460: Create a complementary non-split-join...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/940#discussion_r172373203 --- Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java --- @@ -0,0 +1,415 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.enrichment.bolt; + +import org.apache.metron.common.Constants; +import org.apache.metron.common.bolt.ConfiguredEnrichmentBolt; +import org.apache.metron.common.configuration.ConfigurationType; +import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig; +import org.apache.metron.common.error.MetronError; +import org.apache.metron.common.performance.PerformanceLogger; +import org.apache.metron.common.utils.ErrorUtils; +import org.apache.metron.common.utils.MessageUtils; +import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase; +import org.apache.metron.enrichment.configuration.Enrichment; +import org.apache.metron.enrichment.interfaces.EnrichmentAdapter; +import org.apache.metron.enrichment.parallel.EnrichmentContext; +import org.apache.metron.enrichment.parallel.EnrichmentStrategies; +import org.apache.metron.enrichment.parallel.ParallelEnricher; +import org.apache.metron.enrichment.parallel.WorkerPoolStrategy; +import org.apache.metron.stellar.dsl.Context; +import org.apache.metron.stellar.dsl.StellarFunction; +import org.apache.metron.stellar.dsl.StellarFunctions; +import org.apache.storm.task.OutputCollector; +import org.apache.storm.task.TopologyContext; +import org.apache.storm.topology.OutputFieldsDeclarer; +import org.apache.storm.tuple.Fields; +import org.apache.storm.tuple.Tuple; +import org.apache.storm.tuple.Values; +import org.json.simple.JSONObject; +import org.json.simple.parser.JSONParser; +import org.json.simple.parser.ParseException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.UnsupportedEncodingException; +import java.lang.invoke.MethodHandles; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.UUID; + +/** + * This bolt is a unified enrichment/threat intel bolt. In contrast to the split/enrich/join + * bolts above, this handles the entire enrichment lifecycle in one bolt using a threadpool to + * enrich in parallel. + * + * From an architectural perspective, this is a divergence from the polymorphism based strategy we have + * used in the split/join bolts. Rather, this bolt is provided a strategy to use, either enrichment or threat intel, + * through composition. This allows us to move most of the implementation into components independent + * from Storm. This will greater facilitate reuse. + */ +public class UnifiedEnrichmentBolt extends ConfiguredEnrichmentBolt { + + public static class Perf {} // used for performance logging + private PerformanceLogger perfLog; // not static bc multiple bolts may exist in same worker + + protected static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass()); + + public static final String STELLAR_CONTEXT_CONF = "stellarContext"; + + /** + * The number of threads in the threadpool. One threadpool is created per process. + * This is a topology-level configuration + */ + public static final String THREADPOOL_NUM_THREADS_TOPOLOGY_CONF = "metron.threadpool.size"; + /** + * The type of threadpool to create. This is a topology-level configuration. + */ + public static final String THREADPOOL_TYPE_TOPOLOGY_CONF = "metron.threadpool.type"; + + /** + * The enricher implementation to use. This will do the parallel enrichment via a thread pool. + */ + protected ParallelEnricher enricher; + + /** + * The strategy to use for this enrichment bolt. Practically speak
[GitHub] metron pull request #940: METRON-1460: Create a complementary non-split-join...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/940#discussion_r172369461 --- Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/EnrichmentStrategies.java --- @@ -0,0 +1,79 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.enrichment.parallel; + +import org.apache.metron.common.Constants; +import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig; +import org.apache.metron.common.configuration.enrichment.handler.ConfigHandler; +import org.apache.metron.enrichment.bolt.CacheKey; +import org.json.simple.JSONObject; +import org.slf4j.Logger; + +import java.util.Map; +import java.util.concurrent.Executor; + +public enum EnrichmentStrategies implements Strategy { --- End diff -- This is a strategy pattern using an enum. The purpose of this class is to resolve the specific strategies possible. It's broadly in line with other strategy patterns (e.g. Extractors). ---
[GitHub] metron pull request #940: METRON-1460: Create a complementary non-split-join...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/940#discussion_r172369480 --- Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/Strategy.java --- @@ -0,0 +1,47 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.enrichment.parallel; + +import org.apache.metron.common.Constants; +import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig; +import org.apache.metron.common.configuration.enrichment.handler.ConfigHandler; +import org.json.simple.JSONObject; +import org.slf4j.Logger; + +import java.util.Map; + +/** + * Enrichment strategy. This interface provides a mechanism to interface with the enrichment config and any + * post processing steps that are needed to be done after-the-fact. + * + * The reasoning behind this is that the key difference between enrichments and threat intel is that they pull + * their configurations from different parts of the SensorEnrichmentConfig object and as a post-join step, they differ + * slightly. + * + */ +public interface Strategy { + Constants.ErrorType getErrorType(); --- End diff -- Sure thing. ---
[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/940 Ahhh, that makes sense. I bet we were getting killed by small allocations in the caching layer. ---
[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/940 I actually suspect GC as well. We adjusted the garbage collector to the G1GC and saw throughput gains, but not nearly the kinds of gains as we got with a drop-in of Caffeine to replace Guava. ---
[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/940 In this case, the loader isn't doing anything terribly expensive, though it may in the future (incur a hbase get or some more expensive computation). ---
[GitHub] metron issue #924: METRON-1299 In MetronError tests, don't test for HostName...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/924 +1, sorry! ---
[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/940 We actually did increase the concurrency level for guava to 64; that is what confused us as well. The hash code is mostly standard, should be evenly distributed (the key is pretty much a POJO). ---
[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/940 The interesting thing that we found was that guava seems to be doing poorly when the # of items in the cache gets large. When we scaled the test down (830 distinct IP addresses chosen randomly and sent in at a rate of 200k events per second with a cache size of 100) kept up but scaling the test up (300k distinct ip addresses chosen randomly and sent in at a rate of 200k events per second with a cache size of 100k) didn't. ---
[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/940 We were being purposefully unkind to the cache in the tests. The load simulation chose a IP address at random to present, so each IP had an equal probability of being selected. Whereas, in real traffic, we expect a coherent working set. Not sure of the exact hit rates, though. ---
[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/940 I ran this up with vagrant and ensured: * Normal stellar works still in field transformations as well as enrichments * swapped in and out new enrichments live * swapped in and out new threat intel live Are there any other pending issues here beyond a report of the performance impact? ---
[GitHub] metron issue #944: METRON-1463: Adjust the groupings and shuffles in enrichm...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/944 I ran this up with vagrant and ensured: * Normal stellar works still in field transformations as well as enrichments * swapped in and out new enrichments live * swapped in and out new threat intel live ---
[GitHub] metron issue #947: METRON-1467: Replace guava caches in places where the key...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/947 I ran this up with vagrant and ensured: * Normal stellar works still in field transformations as well as enrichments * swapped in and out new enrichments live * swapped in and out new threat intel live ---
[GitHub] metron pull request #947: METRON-1467: Replace guava caches in places where ...
GitHub user cestella reopened a pull request: https://github.com/apache/metron/pull/947 METRON-1467: Replace guava caches in places where the keyspace might be large (NOTE: Review after METRON-1460) ## Contributor Comments Based on the performance tuning exercise as part of METRON-1460, guava has difficulties with cache sizes over 10k. We, unfortunately, are quite demanding of guava in this regard so we should transition a few uses of guava to Caffeine: * Stellar processor cache * The JoinBolt cache * The Enrichment Bolt Cache NOTE: This depends on METRON-1460 aka #940 and that PR is included in here. It might be easier to review after #940 is merged. Test plan pending ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron guava_cache_replacement Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/947.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #947 commit a4f618a3ad895d62772366e0e93e5b8b37c5c964 Author: cstella <cestella@...> Date: 2018-02-21T23:59:16Z Adding parallel enrichment bolt. commit 99fe0b86005fe04294b3851a17ae3d88f228c5d2 Author: cstella <cestella@...> Date: 2018-02-22T00:21:06Z Updating to include trace statements. commit 79736c6f3fab04d01dd1eb998b308f438003a0e1 Author: cstella <cestella@...> Date: 2018-02-22T15:35:44Z Updating with some cleanup commit cb4a527c9146865dafad1d597ba93032ef398d94 Author: cstella <cestella@...> Date: 2018-02-22T15:48:11Z Updating spec. commit fb4d4383f366776f446e33a422652c3ec1f56bfa Author: cstella <cestella@...> Date: 2018-02-22T18:00:36Z Updating threadpool creation commit 87ef6a72827c31f8adee42ee71272a32c350bc1f Author: cstella <cestella@...> Date: 2018-02-22T18:04:37Z better docs commit 6ae9594ee4ae2b4d33e0feca398b527077dac0d3 Author: cstella <cestella@...> Date: 2018-02-22T18:41:20Z Updating readme. commit 82ebc9550d759ea0bd06b48c586
[GitHub] metron pull request #947: METRON-1467: Replace guava caches in places where ...
Github user cestella closed the pull request at: https://github.com/apache/metron/pull/947 ---
[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/940 Just FYI, as part of the performance experimentation in the lab here, we found that one major impediment to scale was the guava cache in this topology when the size of the cache becomes non-trivial in size (e.g. 10k+). Swapping out [Caffeine](https://github.com/ben-manes/caffeine) immediately had a substantial affect. I created #947 to migrate the split/join infrastructure to use caffeine as well and will look at the performance impact of that change. I wanted to separate that work from here as it may be that guava performance is fine outside of an explicit threadpool like we have here. ---
[GitHub] metron pull request #947: METRON-1467: Replace guava caches in places where ...
GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/947 METRON-1467: Replace guava caches in places where the keyspace might be large ## Contributor Comments Based on the performance tuning exercise as part of METRON-1460, guava has difficulties with cache sizes over 10k. We, unfortunately, are quite demanding of guava in this regard so we should transition a few uses of guava to Caffeine: * Stellar processor cache * The JoinBolt cache * The Enrichment Bolt Cache NOTE: This depends on METRON-1460 aka #940 Test plan pending ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron guava_cache_replacement Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/947.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #947 commit a4f618a3ad895d62772366e0e93e5b8b37c5c964 Author: cstella <cestella@...> Date: 2018-02-21T23:59:16Z Adding parallel enrichment bolt. commit 99fe0b86005fe04294b3851a17ae3d88f228c5d2 Author: cstella <cestella@...> Date: 2018-02-22T00:21:06Z Updating to include trace statements. commit 79736c6f3fab04d01dd1eb998b308f438003a0e1 Author: cstella <cestella@...> Date: 2018-02-22T15:35:44Z Updating with some cleanup commit cb4a527c9146865dafad1d597ba93032ef398d94 Author: cstella <cestella@...> Date: 2018-02-22T15:48:11Z Updating spec. commit fb4d4383f366776f446e33a422652c3ec1f56bfa Author: cstella <cestella@...> Date: 2018-02-22T18:00:36Z Updating threadpool creation commit 87ef6a72827c31f8adee42ee71272a32c350bc1f Author: cstella <cestella@...> Date: 2018-02-22T18:04:37Z better docs commit 6ae9594ee4ae2b4d33e0feca398b527077dac0d3 Author: cstella <cestella@...> Date: 2018-02-22T18:41:20Z Updating readme. commit 82ebc9550d759ea0bd06b48c586fd5e53c6e553a Author: cstella <cestella@...> Date: 2018-02-22T20:53:31Z Better documentation. commit 235046d3d1
[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/940 @arunmahadevan Thanks for chiming in Arun. I would say that most of the enrichment work is I/O bound and we try to avoid it whenever possible with a a time-evicted LRU cache in front of the enrichments. We don't always know a priori what enrichments users are doing, per se, as their individual enrichments may be expressed via stellar. The threads here are entirely managed via the fixed threadpool service in storm and the threadpool is shared across all of the executors running in-process on the worker, so we try to minimize that. ---
[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/940 @ottobackwards I haven't sent an email to the storm team, but I did run the PR past a storm committer that I know and asked his opinion prior to submitting the PR. The general answer was something to the effect of `The overall goal should be to reduce the network shuffle unless its really required.` Also, the notion of using an external threadpool didn't seem to be fundamentally offensive. ---
[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/940 @mraliagha It's definitely a tradeoff. This is why this is as a complement to the original split/join topology. Keep in mind, also, that this architecture enables use-cases that the other would prevent or make extremely difficult and/or network intensive, such as multi-level stellar statements rather than the 2 levels we have now. We are undergoing some preliminary testing in-lab right now, which @nickwallen alluded to, to compare the two approaches under at least synthetic load and will report back. Ultimately this boils down to efficiencies gained by avoiding network hops and whether that's going to provide an outsized impact, I think. ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user cestella commented on the issue: https://github.com/apache/metron/pull/946 ah, crap, looked at the wrong setting. That's what I meant instead of `storm.library.path` ---
[GitHub] metron pull request #946: METRON-1465:Support for Elasticsearch X-pack
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/946#discussion_r171624949 --- Diff: metron-platform/elasticsearch-shaded/pom.xml --- @@ -31,7 +43,7 @@ org.elasticsearch.client -transport +x-pack-transport --- End diff -- Yep, I concede the point, that is not appropriately licensed via apache and we cannot bundle that dependency. ---
[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack
Github user cestella commented on the issue: https://github.com/apache/metron/pull/946 @mmiklavc Yeah, I think that's the approach, however, there's a snag. Storm requires us to create uber jars, so probably what we want to do is have users actually put the xpath transport client on the storm.library.path. ---
[GitHub] metron pull request #946: METRON-1465:Support for Elasticsearch X-pack
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/946#discussion_r171618015 --- Diff: metron-platform/elasticsearch-shaded/pom.xml --- @@ -31,7 +43,7 @@ org.elasticsearch.client -transport +x-pack-transport --- End diff -- I believe this is dual licensed ES commercial and ASLv2. IT comes from [here](https://github.com/elastic/elasticsearch#license). ---
[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/940 @nickwallen Sounds good. When scale tests are done, can we make sure that we also include #944 ? ---
[GitHub] metron issue #938: METRON-1457: Move ASF links to main page in the Metron we...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/938 +1 as well, looks great ---
[GitHub] metron issue #938: METRON-1457: Move ASF links to main page in the Metron we...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/938 Where is that powered by apache logo from? Are we sure it doesn't mean that the apache web server serves it up? ---
[GitHub] metron pull request #944: METRON-1463: Adjust the groupings and shuffles in ...
GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/944 METRON-1463: Adjust the groupings and shuffles in enrichment to be more efficient ## Contributor Comments Currently there are some deficiencies in our grouping approach in the enrichment topology: * We have field grouping by key in places where we need LOCAL_OR_SHUFFLE groupings * We have shuffle groupings in places where we need LOCAL_OR_SHUFFLE groupings * We have field groupings by key in places where we need field grouping by message (specifically in the connections from the splitter to the enrichments). * Field grouping by key, as we had before, ensured that we had fewer cache hits as opposed to field grouping by message. To test this, just run things up in full-dev and ensure that data continues to flow. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron better_grouping_in_splitjoin Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/944.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #944 commit dd8821af912071e357c42c852361292c7c2c655c Author: cstella <cestella@...> Date: 2018-02-27T14:28:00Z Adjusted groupings to be local or shuffle ---
[GitHub] metron issue #934: METRON-1423: Ambari work to handle Solr configuration
Github user cestella commented on the issue: https://github.com/apache/metron/pull/934 +1 by inspection ---
[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/940 The current architecture is described ![Image of Yaktocat](https://github.com/apache/metron/raw/master/metron-platform/metron-enrichment/enrichment_arch.png) In short, for each message each splitter will * inspect the configs for the sensor * For each sensor, extract the fields required for enrichment and send them to the appropriate enrichment bolt (e.g. hbase, geo, stellar) * If one enrichment enriches k fields, then k messages will be sent to the enrichment bolt * In the case of stellar, each stellar subgroup will be a separate message * the original message is sent directly to the join bolt * The enrichment bolts do the enrichment and send the additional fields and values to the original message * The join bolt will asynchronously collect the subresults and join them with the original message * The join bolt has a LRU cache to hold subresults until all results arrive * Tuning performance involves tuning this cache (max size and time until eviction) * Tuning this can be complex because it has to be large enough to handle spikes in traffic ---
[GitHub] metron issue #933: METRON-1452 Rebase Dev Environment on Latest CentOS 6
Github user cestella commented on the issue: https://github.com/apache/metron/pull/933 piling on, +1 by inspection ---
[GitHub] metron issue #934: METRON-1423: Ambari work to handle Solr configuration
Github user cestella commented on the issue: https://github.com/apache/metron/pull/934 Chiming in late. I agree that we should not have an explicit dependency on an indexing Mpack, even one not of our own construction. I think people will have a lot of different ways to install solr and Metron's mpack should just be configured to point to an existing solr instance. I would generally be in favor of adding support for: * solr.commitPerBatch * solr.commit.waitSearcher * solr.commit.waitFlush * solr.commit.soft * solr.collection * solr.http.config but sensible defaults are chosen there and people can adjust them in the global config, so I think they can wait for a follow-on. Some of them may be problematic to encode in a UI (solr.http.config is a map, for instance), but most of them would be pretty trivial. Frankly, I'm a bit hesitant to give people the ability to screw with transactions details easily. ---
[GitHub] metron pull request #934: METRON-1423: Ambari work to handle Solr configurat...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/934#discussion_r170269639 --- Diff: metron-platform/metron-solr/src/main/scripts/create_collection.sh --- @@ -0,0 +1,27 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +METRON_VERSION=${project.version} +METRON_HOME=/usr/metron/$METRON_VERSION +SOLR_VERSION=6.6.2 --- End diff -- I notice that we're depending on maven artifacts of 6.6.0 and we set the solr version to 6.6.2 here, is there a reason for that? If we can align them, then we could replace this line with SOLR_VERSION=`${global_solr_version}` This question applies for the rest of the shell scripts too ---
[GitHub] metron pull request #940: Single bolt split join poc
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/940#discussion_r170089790 --- Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java --- @@ -0,0 +1,323 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.enrichment.bolt; + +import org.apache.metron.common.Constants; +import org.apache.metron.common.bolt.ConfiguredEnrichmentBolt; +import org.apache.metron.common.configuration.ConfigurationType; +import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig; +import org.apache.metron.common.configuration.enrichment.handler.ConfigHandler; +import org.apache.metron.common.error.MetronError; +import org.apache.metron.common.performance.PerformanceLogger; +import org.apache.metron.common.utils.ErrorUtils; +import org.apache.metron.common.utils.MessageUtils; +import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase; +import org.apache.metron.enrichment.configuration.Enrichment; +import org.apache.metron.enrichment.interfaces.EnrichmentAdapter; +import org.apache.metron.enrichment.parallel.EnrichmentContext; +import org.apache.metron.enrichment.parallel.EnrichmentStrategies; +import org.apache.metron.enrichment.parallel.ParallelEnricher; +import org.apache.metron.enrichment.parallel.WorkerPoolStrategy; +import org.apache.metron.stellar.dsl.Context; +import org.apache.metron.stellar.dsl.StellarFunction; +import org.apache.metron.stellar.dsl.StellarFunctions; +import org.apache.storm.task.OutputCollector; +import org.apache.storm.task.TopologyContext; +import org.apache.storm.topology.OutputFieldsDeclarer; +import org.apache.storm.tuple.Fields; +import org.apache.storm.tuple.Tuple; +import org.apache.storm.tuple.Values; +import org.json.simple.JSONObject; +import org.json.simple.parser.JSONParser; +import org.json.simple.parser.ParseException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.UnsupportedEncodingException; +import java.lang.invoke.MethodHandles; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.UUID; + --- End diff -- Ok, I went through and did more rigorous documentation throughout the new classes. Let me know if it makes sense or if there are still issues. ---
[GitHub] metron pull request #940: Single bolt split join poc
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/940#discussion_r170057327 --- Diff: metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java --- @@ -0,0 +1,323 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.enrichment.bolt; + +import org.apache.metron.common.Constants; +import org.apache.metron.common.bolt.ConfiguredEnrichmentBolt; +import org.apache.metron.common.configuration.ConfigurationType; +import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig; +import org.apache.metron.common.configuration.enrichment.handler.ConfigHandler; +import org.apache.metron.common.error.MetronError; +import org.apache.metron.common.performance.PerformanceLogger; +import org.apache.metron.common.utils.ErrorUtils; +import org.apache.metron.common.utils.MessageUtils; +import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase; +import org.apache.metron.enrichment.configuration.Enrichment; +import org.apache.metron.enrichment.interfaces.EnrichmentAdapter; +import org.apache.metron.enrichment.parallel.EnrichmentContext; +import org.apache.metron.enrichment.parallel.EnrichmentStrategies; +import org.apache.metron.enrichment.parallel.ParallelEnricher; +import org.apache.metron.enrichment.parallel.WorkerPoolStrategy; +import org.apache.metron.stellar.dsl.Context; +import org.apache.metron.stellar.dsl.StellarFunction; +import org.apache.metron.stellar.dsl.StellarFunctions; +import org.apache.storm.task.OutputCollector; +import org.apache.storm.task.TopologyContext; +import org.apache.storm.topology.OutputFieldsDeclarer; +import org.apache.storm.tuple.Fields; +import org.apache.storm.tuple.Tuple; +import org.apache.storm.tuple.Values; +import org.json.simple.JSONObject; +import org.json.simple.parser.JSONParser; +import org.json.simple.parser.ParseException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.UnsupportedEncodingException; +import java.lang.invoke.MethodHandles; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.UUID; + --- End diff -- Yeah, good call. ---
[GitHub] metron pull request #940: Single bolt split join poc
GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/940 Single bolt split join poc ## Contributor Comments There are some deficiencies to the split/join topology. It's hard to reason about * Understanding the latency of enriching a message requires looking at multiple bolts that each give summary statistics * The join bolt's cache is really hard to reason about when performance tuning * During spikes in traffic, you can overload the join bolt's cache and drop messages if you aren't careful * In general, it's hard to associate a cache size and a duration kept in cache with throughput and latency * There are a lot of network hops per message * Right now we are stuck at 2 stages of transformations being done (enrichment and threat intel). It's very possible that you might want stellar enrichments to depend on the output of other stellar enrichments. In order to implement this in split/join you'd have to create a cycle in the storm topology I propose that we move to a model where we do enrichments in a single bolt in parallel using a static threadpool (e.g. multiple workers in the same process would share the threadpool). IN all other ways, this would be backwards compatible. A transparent drop-in for the existing enrichment topology. There are some pros/cons about this too: * Pro * Easier to reason about from an individual message perspective * Architecturally decoupled from Storm * This sets us up if we want to consider other streaming technologies * Fewer bolts * spout -> enrichment bolt -> threatintel bolt -> output bolt * Way fewer network hops per message currently 2n+1 where n is the number of enrichments used (if using stellar subgroups, each subgroup is a hop) * Easier to reason about from a performance perspective * We trade cache size and eviction timeout for threadpool size * We set ourselves up to have stellar subgroups with dependencies i.e. stellar subgroups that depend on the output of other subgroups If we do this, we can shrink the topology to just spout -> enrichment/threat intel -> output * Con * We can no longer tune stellar enrichments independent from HBase enrichments * To be fair, with enrichments moving to stellar, this is the case in the split/join approach too * No idea about performance What I propose is to submit a PR that will deliver an alternative, completely backwards compatible topology for enrichment that you can use by adjusting the `start_enrichment_topology.sh` script to use `remote-unified.yaml` instead of `remote.yaml`. If we live with it for a while and have some good experiences with it, maybe we can consider retiring the old enrichment topology. To test this, spin up vagrant and edit `$METRON_HOME/bin/start_enrichment_topology.sh` to use `remote-unified.yaml` instead of `remote.yaml`. Restart enrichment and you should see a topology that looks something like: ![image](https://user-images.githubusercontent.com/540359/36556636-e0ae092e-17d3-11e8-9e45-5160b4f23451.png) ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If ad
[GitHub] metron issue #935: METRON-1386: Fix Metron Website Required Links
Github user cestella commented on the issue: https://github.com/apache/metron/pull/935 This looks great! +1 by inspection. ---
[GitHub] metron issue #937: METRON-1455: Patch and Replace methods in the REST Update...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/937 I verified this in full dev with kerberos as well. ---
[GitHub] metron pull request #937: METRON-1455: Patch and Replace methods in the REST...
GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/937 METRON-1455: Patch and Replace methods in the REST UpdateController return 400 ## Contributor Comments Shading and relocating the jackson library had the unexpected behavior of breaking patch requests in the REST API. The solution is that we moved the remaining patch functionality inside `JSONUtils` and stop relying on Jackson specific classes in our external API (e.g. `PatchRequest`). To exercise this behavior, from the alerts UI, click on an message and click "Escalate." From the console ensure that the patch request returns a 200 code rather than a 400 or 500 code. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron METRON-1455 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/937.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #937 commit cce8fda5557d9360e43de8a12443fa67d79e3681 Author: cstella <cestella@...> Date: 2018-02-14T19:15:38Z METRON-1455: Patch and Replace methods in the REST UpdateController return 400 ---
[GitHub] metron pull request #929: METRON-1448: Update SolrWriter to conform to new c...
Github user cestella closed the pull request at: https://github.com/apache/metron/pull/929 ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/929 I updated the instructions and the `solr.zookeeper` business is legacy naming, but I think it does fit since it's specific to zookeeper rather than connecting to ES master. I dunno, that's just what I told myself ;) ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/929 Ok, I wrote up a detailed testing plan to get this running in full-dev. I want to add a couple of unit tests around the new properties that I added, but at that point, this is feature complete and ready to review. ---
[GitHub] metron pull request #929: METRON-1448: Update SolrWriter to conform to new c...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/929#discussion_r166798136 --- Diff: metron-platform/metron-solr/src/main/java/org/apache/metron/solr/writer/SolrWriter.java --- @@ -33,17 +39,19 @@ import java.io.IOException; import java.io.Serializable; -import java.util.List; -import java.util.Map; +import java.lang.invoke.MethodHandles; +import java.util.*; --- End diff -- UGh, it slipped back in again. Ok, removing it. ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/929 Ok, I updated the configs to expose all the settings that I could find for committing and documented them. By default we operate safely, though. Though now you can use soft commits rather than durable commits if you so desire and wish to live dangerously. ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/929 LOL, nah, I was just being crotchety. Let me see what I can do and you tell me what you think :) ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/929 Actually, I can make an option to let you do a soft commit instead of a hard commit if that'll be an extra layer of configuration. ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/929 You can set it to not manually commit (set `solr.commitPerBatch` to `false` and no committing happens), but you're risking losing data if a worker dies. Honestly, want a durable commit with a fsync before you ack the tuples in a batch, otherwise you're courting data loss. This is the same strategy we do for ES and HDFS (though commit there is a fsync). That being said, I'm sensitive to performance issues around that that people may have, so I let people turn it off with a strong warning in the docs (also this was legacy behavior in the SolrWriter). ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/929 What do you mean "auto-commit"? By default it will commit at the batch level as the other writers do. You can turn that behavior off and it'll commit when the Solr client decides at the cost of possibly losing data. Regarding solr vs solrcloud, to my understanding solr is the product and solrcloud is a way of running Solr. The other way is solr standalone. Originally we named things after the product, which I think is correct, and have made explicit that we support Solr running in Solr Cloud mode in the docs. ---
[GitHub] metron pull request #929: METRON-1448: Update SolrWriter to conform to new c...
Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/929#discussion_r166682417 --- Diff: metron-platform/metron-solr/src/main/java/org/apache/metron/solr/writer/SolrWriter.java --- @@ -33,17 +39,19 @@ import java.io.IOException; import java.io.Serializable; -import java.util.List; -import java.util.Map; +import java.lang.invoke.MethodHandles; +import java.util.*; --- End diff -- You got it :) ---
[GitHub] metron pull request #929: METRON-1448: Update SolrWriter to conform to new c...
GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/929 METRON-1448: Update SolrWriter to conform to new collection strategy ## Contributor Comments Currently the SolrWriter presumes a single collection to be written to. The new collection strategy for Solr implies a collection per sensor. Also, there are a few rough edges in the writer which could stand smoothing: * By default, we use solr's implicit commit mechanism, rather than committing at the batch granularity. This may result in lost data on worker failure. * We do not use the the batch add api, but rather message-by-message add Testing plan pending. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron SOLR_writer_mod Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/929.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #929 commit 6bb30af9d2005414e3ee44c0bdb0ea14540ce13c Author: cstella <cestella@...> Date: 2018-02-01T21:33:56Z METRON-1441: Create complementary Solr schemas for the main sensors commit f4ff0c401eff23d9c1b2ca3b264bd9b0d4e8f381 Author: cstella <cestella@...> Date: 2018-02-01T21:47:12Z Updating dao commit 7e2ecb0f2f55ea16529128fec14920bc2a546b07 Author: cstella <cestella@...> Date: 2018-02-02T21:43:38Z Migrated data to files, renamed test and added yaf and error. commit 2aacd202ff1a2ebcbeb30300b30d080391cfe1cf Author: cstella <cestella@...> Date: 2018-02-02T21:45:08Z Merge branch 'feature/METRON-1416-upgrade-solr' into SOLR_METRON-1441 commit 2e32e7ea4ef8cace764394c1dec693d8385a6b9a Author: cstella <cestella@...> Date: 2018-02-02T21:50:06Z Added to readme. commit e2901d4bd4b9787f668c2dccd2e4f8aa53a926d7 Author: cstella <cestella@...> Date: 2018-02-05T14:39:31Z Updating error to have a guid and removed docValues=true for bytes type. commit 3c4319ec4581fdb259a697b548a267225316874a Author: cstella <cestella@...>
[GitHub] metron pull request #922: METRON-1441: Create complementary Solr schemas for...
Github user cestella closed the pull request at: https://github.com/apache/metron/pull/922 ---
[GitHub] metron issue #922: METRON-1441: Create complementary Solr schemas for the ma...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/922 @simonellistonball Yes, we should. I added the relevant context and grouping for each of the schemas that we ship by default. Bro has more context as there were more comments in the ES schema. I brought over whatever comments were relevant and grouped appropriately and as best I could in the others. Let me know if I missed anything. ---