[GitHub] incubator-metron issue #432: METRON-682: Unify and Improve the Flat File Loa...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/432 I am trying to test this out - my vm is not doing well at the moment however. Hopefully I can get it straight --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #432: METRON-682: Unify and Improve the Flat File Loa...
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/432
[vagrant@node1 tmp]$ /usr/metron/0.3.0/bin/flatfile_loader.sh -i
http://s3.amazonaws.com/alexa-static/top-1m.csv.zip -t enrichment -c t -e
./extractor.json -p 5 -b 128
Exception in thread "main" org.codehaus.jackson.map.JsonMappingException:
Can not instantiate value of type [map type; class java.util.LinkedHashMap,
[simple type, class java.lang.String] -> [simple type, class java.lang.Object]]
from JSON String; no single-String constructor/factory method (through
reference chain:
org.apache.metron.dataloads.extractor.ExtractorHandler["config"])
at
org.codehaus.jackson.map.deser.std.StdValueInstantiator._createFromStringFallbacks(StdValueInstantiator.java:379)
at
org.codehaus.jackson.map.deser.std.StdValueInstantiator.createFromString(StdValueInstantiator.java:268)
at
org.codehaus.jackson.map.deser.std.MapDeserializer.deserialize(MapDeserializer.java:244)
at
org.codehaus.jackson.map.deser.std.MapDeserializer.deserialize(MapDeserializer.java:33)
at
org.codehaus.jackson.map.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:299)
at
org.codehaus.jackson.map.deser.SettableBeanProperty$MethodProperty.deserializeAndSet(SettableBeanProperty.java:414)
at
org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:697)
at
org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580)
at
org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2732)
at
org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1909)
at
org.apache.metron.dataloads.extractor.ExtractorHandler.load(ExtractorHandler.java:70)
at
org.apache.metron.dataloads.extractor.ExtractorHandler.load(ExtractorHandler.java:75)
at
org.apache.metron.dataloads.extractor.ExtractorHandler.load(ExtractorHandler.java:78)
at
org.apache.metron.dataloads.nonbulk.flatfile.SimpleEnrichmentFlatFileLoader.main(SimpleEnrichmentFlatFileLoader.java:49)
at
org.apache.metron.dataloads.nonbulk.flatfile.SimpleEnrichmentFlatFileLoader.main(SimpleEnrichmentFlatFileLoader.java:40)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
[vagrant@node1 tmp]$
with extractor.json of:
```
{
"config" :
"columns" : {
"domain" : 1,
"rank" : 0
}
,"indicator_column" : "domain"
,"type" : "alexa"
,"separator" : ","
},
"extractor" : "CSV"
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #444: METRON-705: Parallelize the build in travis to ...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/444 you bet btw -the build on this pr isn't even going at all --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #444: METRON-705: Parallelize the build in travis to ...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/444 integration tests took 43 minutes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #444: METRON-705: Parallelize the build in travis to ...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/444 #14 passed Elapsed time 43 min 10 sec Total time 1 hr 6 min 21 sec --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #444: METRON-705: Parallelize the build in travis to ...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/444 So to test this - if you have travis set up for your git account, I think you just push it to a branch in your repo and let travis build it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #361: METRON-630 Add tarLongFileMode posix to maven-a...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/361 Thanks for the contribution --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #444: METRON-705: Parallelize the build in travis to ...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/444 #14.1 passed Elapsed time 23 min 28 sec On my second try around 5pm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #461: METRON-733: Remove Geo database from ParserBolt
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/461 Does this mean you can get rid of the local geo database in the parsers resource dir? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/438 That makes sense, thanks for the explanation. I am +1 for having SOLR specific flattening, although documenting well needs to be a priority. Also, if we do this, please log a jira that the flattening in the json mapper needs to be optional by configuration --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/438 let me know which one to comment on! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #462: METRON-734 Builds failing because of MaxMind DB...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/462 +1, build and tests run ( mvn -q clean integration-test install && build_utils/verify_licenses.sh ), on machine that was failing every every time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/455 ok - i'm not building this branch, just what i've seen from other builds --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #455: METRON-720 modify generate-md.sh to re-throw er...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/455 re: "site.xml is now a fully auto-generated file" that would be why it is showing in git as unstaged but modified? Is there a way we can stop that from happening? If it is generated does it need to be in git? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/438 I think what we are saying is 1. I think that is the best way to move forward. Just make sure that this commit isn't merged into an RC for 0.3.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #449: METRON-701 Triage Metrics Produced by the Profi...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/449 I think routing from stellar within routing in storm is confusing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #449: METRON-701 Triage Metrics Produced by the Profi...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/449 I'm sorry, no Nick I don't --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #450: METRON-690: Create a DSL-based timestamp lookup...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/450 Honestly, I was thinking more about the 'kill everything' part, which seems to be repeated a lot --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #457: METRON-724 add [IN] to the documented ke...
GitHub user ottobackwards opened a pull request: https://github.com/apache/incubator-metron/pull/457 METRON-724 add [IN] to the documented keywords that require escaping The keyword 'in' must be escaped to be used in Stellar, but is not documented as such. This PR adds in the documentation and a unit test. Verify: - by running the tests - by using the Stellar Console to evaluate an expression - by building and viewing the documentation ( see below ) and the readme.md in github Thank you for submitting a contribution to Apache Metron (Incubating). Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification guildlines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ n/a] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x ] Have you included steps or a guide to how the change may be verified and tested manually? - [x ] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [x ] Have you written or updated unit tests and or integration tests to verify your changes? - [n/a ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [n/a ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html. ``` cd site-book bin/generate-md.sh mvn site:site ``` ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommened that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/incubator-metron METRON-724 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/457.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #457 commit b90987bbeca346301a18aea2a7913a072fde4a3a Author: Otto Fowler <ottobackwa...@gmail.com> Date: 2017-02-17T17:06:58Z METRON-724 add [IN] to the documented keywords that require escaping add in unit test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #461: METRON-733: Remove Geo database from ParserBolt
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/461 I am seeing this on my local branch, not the PR On February 22, 2017 at 09:13:12, Nick Allen (notificati...@github.com) wrote: The build error does not seem related to your PR. I saw the same failure on #450 <https://github.com/apache/incubator-metron/pull/450>. The resource com.fasterxml:oss-parent:pom:28 really does not exist in Maven Central, AFAIK. Hmm. [ERROR] Failed to execute goal on project metron-enrichment: Could not resolve dependencies for project org.apache.metron:metron-enrichment:jar:0.3.1: Failed to collect dependencies at com.maxmind.geoip2:geoip2:jar:2.8.0 -> com.maxmind.db:maxmind-db:jar:1.2.1 -> com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failed to read artifact descriptor for com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Could not find artifact com.fasterxml:oss-parent:pom:28 in central (http://repo.maven.apache.org/maven2) -> [Help 1] â You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/apache/incubator-metron/pull/461#issuecomment-281679880>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABD1_98dtsK9jFWAsUhiM3fkRJZzHr_Kks5rfEJ3gaJpZM4MIH4d> . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/438 I commented on 735 so it wins. #settled --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/438 Nick - if you look at the JSONMapParser, Casey and I implemented a flattener that you would use for this. So, all of our json stuff from that _was_ already flattened. I am more confused about there being *non* flattened things going through actually. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #454: METRON-716 site-book README.md
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/454 @mattf-horton are you ok on this after the revisions? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #454: METRON-716 site-book README.md
GitHub user ottobackwards opened a pull request: https://github.com/apache/incubator-metron/pull/454 METRON-716 site-book README.md Initial draft of a readme for site-book documentation. Questions: should we list the tools used more explicitly? You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/incubator-metron METRON-716 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/454.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #454 commit f56190441f4751bc2493b49b1495258ed0a25c5b Author: Otto Fowler <ottobackwa...@gmail.com> Date: 2017-02-13T20:54:26Z initial draft of site-book README.md --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #452: Removed MySQL from Enrichment Diagram
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/452 Please rename the PR to start with "METRON-715". Then we can land this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #453: METRON-694: Index Errors from Topologies
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/453 Just real quick before I start to review: - What are the performance implications of this? How can we measure that? - What is the effect of running this on quick dev? is it to be enabled by default or disabled for load/performance? - Does this run in Docker? - I don't see any documentation changes. What needs to be done, should it be in this or another PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #453: METRON-694: Index Errors from Topologies
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/453#discussion_r100815725
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/error/MetronError.java
---
@@ -0,0 +1,200 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.error;
+
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.lang.exception.ExceptionUtils;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.Constants.ErrorType;
+import org.json.simple.JSONObject;
+
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.metron.common.Constants.ErrorFields;
+
+public class MetronError {
+
+ private String message;
+ private Throwable throwable;
+ private String sensorType = "error";
+ private ErrorType errorType = ErrorType.DEFAULT_ERROR;
+ private Set errorFields;
+ private List rawMessages;
+
+ public MetronError withMessage(String message) {
+this.message = message;
+return this;
+ }
+
+ public MetronError withThrowable(Throwable throwable) {
+this.throwable = throwable;
+return this;
+ }
+
+ public MetronError withSensorType(String sensorType) {
+this.sensorType = sensorType;
+return this;
+ }
+
+ public MetronError withErrorType(ErrorType errorType) {
+this.errorType = errorType;
+return this;
+ }
+
+ public MetronError withErrorFields(Set errorFields) {
+this.errorFields = errorFields;
+return this;
+ }
+
+
+ public MetronError addRawMessage(Object rawMessage) {
+if (rawMessage != null) {
+ if (this.rawMessages == null) {
+this.rawMessages = new ArrayList<>();
+ }
+ this.rawMessages.add(rawMessage);
+}
+return this;
+ }
+
+ public MetronError withRawMessages(List rawMessages) {
+this.rawMessages = rawMessages;
+return this;
+ }
+
+ public Optional getThrowable() {
+return throwable != null ? Optional.of(throwable) : Optional.empty();
+ }
+
+ @SuppressWarnings({"unchecked"})
+ public JSONObject getJSONObject() {
+JSONObject errorMessage = new JSONObject();
+errorMessage.put(Constants.SENSOR_TYPE, "error");
+
+ /*
+ * Save full stack trace in object.
+*/
+if (throwable != null) {
+ String stackTrace = ExceptionUtils.getStackTrace(throwable);
+ String exception = throwable.toString();
+ errorMessage.put(ErrorFields.EXCEPTION.getName(), exception);
+ errorMessage.put(ErrorFields.STACK.getName(), stackTrace);
+}
+
+errorMessage.put(ErrorFields.TIMESTAMP.getName(),
System.currentTimeMillis());
+try {
+ errorMessage.put(ErrorFields.HOSTNAME.getName(),
InetAddress.getLocalHost().getHostName());
+} catch (UnknownHostException ex) {
+
+}
+if(rawMessages != null) {
+ for(int i = 0; i < rawMessages.size(); i++) {
+Object rawMessage = rawMessages.get(i);
+// If multiple messages are included add an index to the field
name, otherwise leave it off
+String rawMessageField = rawMessages.size() == 1 ?
ErrorFields.RAW_MESSAGE.getName() : ErrorFields.RAW_MESSAGE.getName() + "_" + i;
+String rawMessageBytesField = rawMessages.size() == 1 ?
ErrorFields.RAW_MESSAGE_BYTES.getName() :
ErrorFields.RAW_MESSAGE_BYTES.getName() + "_" + i;
+if(
[GitHub] incubator-metron pull request #453: METRON-694: Index Errors from Topologies
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/453#discussion_r100815478
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/error/MetronError.java
---
@@ -0,0 +1,200 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.error;
+
+import org.apache.commons.codec.digest.DigestUtils;
+import org.apache.commons.lang.exception.ExceptionUtils;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.Constants.ErrorType;
+import org.json.simple.JSONObject;
+
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.metron.common.Constants.ErrorFields;
+
+public class MetronError {
+
+ private String message;
+ private Throwable throwable;
+ private String sensorType = "error";
+ private ErrorType errorType = ErrorType.DEFAULT_ERROR;
+ private Set errorFields;
+ private List rawMessages;
+
+ public MetronError withMessage(String message) {
+this.message = message;
+return this;
+ }
+
+ public MetronError withThrowable(Throwable throwable) {
+this.throwable = throwable;
+return this;
+ }
+
+ public MetronError withSensorType(String sensorType) {
+this.sensorType = sensorType;
+return this;
+ }
+
+ public MetronError withErrorType(ErrorType errorType) {
+this.errorType = errorType;
+return this;
+ }
+
+ public MetronError withErrorFields(Set errorFields) {
+this.errorFields = errorFields;
+return this;
+ }
+
+
+ public MetronError addRawMessage(Object rawMessage) {
+if (rawMessage != null) {
+ if (this.rawMessages == null) {
+this.rawMessages = new ArrayList<>();
+ }
+ this.rawMessages.add(rawMessage);
+}
+return this;
+ }
+
+ public MetronError withRawMessages(List rawMessages) {
+this.rawMessages = rawMessages;
+return this;
+ }
+
+ public Optional getThrowable() {
+return throwable != null ? Optional.of(throwable) : Optional.empty();
+ }
+
+ @SuppressWarnings({"unchecked"})
+ public JSONObject getJSONObject() {
+JSONObject errorMessage = new JSONObject();
+errorMessage.put(Constants.SENSOR_TYPE, "error");
+
+ /*
+ * Save full stack trace in object.
--- End diff --
Instead of having a method with multiple null checks
Maybe have a method that just calls other methods, and put the checks in
there
handleThrowable(errorMessage);
handleRawMessages(errorMessage);
etc etc
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #453: METRON-694: Index Errors from Topologies
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/453#discussion_r100817317 --- Diff: metron-platform/metron-elasticsearch/src/main/config/elasticsearch_error.properties --- @@ -0,0 +1,69 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + --- End diff -- This is duplicated in the MPACK isn't it? Is there not a way to have this defined once and used in multiple places? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #453: METRON-694: Index Errors from Topologies
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/453#discussion_r100820671
--- Diff:
metron-platform/metron-indexing/src/main/config/zookeeper/indexing/error.json
---
@@ -0,0 +1,17 @@
+{
+ "hdfs" : {
--- End diff --
error is too generic.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #453: METRON-694: Index Errors from Topologies
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/453#discussion_r100813285 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml --- @@ -37,6 +37,12 @@ Metron apps indexed HDFS dir --- End diff -- Should these HDSF directories more specifically pertain to topology processing? As the other use cases came into being, just 'error' may be too generic --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #453: METRON-694: Index Errors from Topologies
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/453 That makes complete sense, we should call that stuff out pre-review. I think what we are seeing is that folks have some really good ideas and are willing to contribute to documentation on this project, we should take advantage. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #453: METRON-694: Index Errors from Topologies
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/453 What comes to mind is the 'source' of an error. Is this error wrong because METRON thinks it is invalid, or it it wrong because of some other configuration specific evaluation. I don't think we do this now with stellar ( IF (BUSINESS_RULE(rule_valid_field,field1) == FALSE) DROP_ERROR()) however. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #453: METRON-694: Index Errors from Topologies
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/453 @cestella - how can we utilize the error indexing if we were going to say - output errors or warnings that there were deprecated stellar statements? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #456: METRON-721 Add github pull request template to ...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/456 I put a reference to the verification doc at the top ' for smoke test guide ' and then put a checkbox for running in vagrant - how's that? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #456: METRON-721 Add github pull request template to ...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/456 So - why don't we come up with a cut down version of that, maybe restricted to trying quick or full dev? or reference and link like we do with the full dev guide? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #456: METRON-721 Add github pull request template to ...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/456 Thanks for the review! I am not sure about verifying builds. I don't think we should put everything that is in the Dev Guide or the Verification in the PR template, there is some balance to be had there. I don't think we expect every PR to have validated the builds anyways. I did make the jira change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #456: METRON-721 Add github pull request templ...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/456#discussion_r101297815 --- Diff: .github/PULL_REQUEST_TEMPLATE.md --- @@ -0,0 +1,40 @@ +Thank you for submitting a contribution to Apache Metron (Incubating). +Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions + +In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check +the following: + +### For all changes: +- [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel) + +- [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. + +- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? + + +### For code changes: +- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? +- [ ] Have you included steps or a guide to how the change may be verified and tested manually? +- [ ] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: + +``` +mvn -q clean integration-test install && build_utils/verify_licenses.sh +``` + +- [ ] Have you written or updated unit tests and or integration tests to verify your changes? +- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? + +### For documentation related changes: +- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? + +``` +cd site-book +bin/generate-md.sh +mvn site:site + +view changes via target/site/index.html --- End diff -- done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #422: METRON-670 Monit Incorrectly Reports Status
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/422 There are also problems with shutting down topologies, since we don't pass in the wait to storm ( at least we didn't. I need to find the pr/jira for that. All external calls from monit need to have clear timeout accounting from both monit's pov and the external agent --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #316: METRON-503: Metron REST API
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/316 Also, the point that this may be better managed from within ambari as a view may need consideration. On January 19, 2017 at 08:58:51, JJ Meyer (notificati...@github.com) wrote: @merrimanr <https://github.com/merrimanr> @cestella <https://github.com/cestella> does it make sense to hold off on this for a little while? It looks like all configuration may be managed by Ambari going forward. I am getting this from the dev thread: Ambari Metron Configuration Management consequences and call to action. â You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/apache/incubator-metron/pull/316#issuecomment-273782615>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABD1_zjUuptzwwa_4XekwsX-AFZ6qGNAks5rT2wZgaJpZM4KbJgW> . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #220: Metron-342 Fix permissions of {{ metron_directo...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/220 Can we close this then? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #467: METRON-743: Sort the files when reading results...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/467 Thank you for clarifying, that makes sense. Sorry to confuse the issue --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #466: METRON-742: Generated code for profile window s...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/466 [INFO] [INFO] Reactor Summary: [INFO] [INFO] Metron . SUCCESS [ 0.002 s] [INFO] metron-analytics ... SUCCESS [ 0.000 s] [INFO] metron-maas-common . SUCCESS [ 0.800 s] [INFO] metron-platform SUCCESS [ 0.001 s] [INFO] metron-test-utilities .. SUCCESS [ 0.357 s] [INFO] metron-integration-test SUCCESS [ 2.171 s] [INFO] metron-maas-service SUCCESS [ 1.720 s] [INFO] metron-common .. SUCCESS [ 40.376 s] [INFO] metron-statistics .. SUCCESS [01:03 min] [INFO] metron-hbase ... SUCCESS [01:11 min] [INFO] metron-profiler-common . SUCCESS [ 10.194 s] [INFO] metron-profiler-client . SUCCESS [01:30 min] [INFO] metron-profiler SUCCESS [ 29.689 s] [INFO] metron-writer .. SUCCESS [ 15.689 s] [INFO] metron-enrichment .. SUCCESS [ 49.701 s] [INFO] metron-indexing SUCCESS [ 11.529 s] [INFO] metron-solr SUCCESS [ 13.183 s] [INFO] metron-pcap SUCCESS [ 0.785 s] [INFO] metron-parsers . SUCCESS [ 48.027 s] [INFO] metron-pcap-backend SUCCESS [ 29.235 s] [INFO] metron-data-management . SUCCESS [01:11 min] [INFO] metron-api . SUCCESS [ 26.331 s] [INFO] metron-management .. SUCCESS [ 39.091 s] [INFO] elasticsearch-shaded ... SUCCESS [ 5.547 s] [INFO] metron-elasticsearch ... SUCCESS [ 41.378 s] [INFO] metron-deployment .. SUCCESS [ 0.000 s] [INFO] Metron Ambari Management Pack .. SUCCESS [ 0.148 s] [INFO] metron-docker .. SUCCESS [ 2.782 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 11:05 min [INFO] Finished at: 2017-02-26T00:05:34-05:00 [INFO] Final Memory: 112M/2199M [INFO] 0:05 ottofowler@Winterfell /Users/ottofowler/src/apache/work/incubator-metron-pr-466 % git status On branch pr-466 nothing to commit, working tree clean +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #467: METRON-743: Sort the files when reading results...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/467 There are two things here I *think* 1. The test assumptions and the failure on some platforms ( if this is indeed the problem ) 2. What this means for the real use of the pcap query system I think we can address the test first, and may have to discuss the 2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #467: METRON-743: Sort the files when reading results...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/467 What are the performance penalties of doing this? Part of the discussion in [HADOOP-12009](https://issues.apache.org/jira/browse/HADOOP-12009) that lead to the clarification of this behavior in the spec was about why they don't do this themselves for the performance penalty. Could you get OOM errors or other issues running large queries that would have worked before? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #466: METRON-742: Generated code for profile window s...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/466 So - the test is - build without changing anything after clone, and there should not be shown in git status right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #466: METRON-742: Generated code for profile window s...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/466 Does building or running the tests have anything to do with it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #468: METRON-744: Allow Stellar functions to be loade...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/468 Run the free up space script --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 ansible-playbook -v playbooks/metron_build.yml Using /Users/ottofowler/src/apache/forks/incubator-metron/metron-deployment/ansible.cfg as config file [WARNING]: provided hosts list is empty, only localhost is available __ < PLAY > -- \ ^__^ \ (oo)\___ (__)\ )\/\ ||w | || || skipping: no hosts matched < PLAY RECAP > \ ^__^ \ (oo)\___ (__)\ )\/\ ||w | || || --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 Do I have to create an inventory with my local machine name? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 I want to run the playbook to just build the rpm's, not deploy. So just metron_build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 How are you running the playbook? I cannot get it to execute --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 can you share your command line? Do you run it from /playbooks? does you -i an inventory? Did you copy or create an ansible.cfg? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 I need to run the playbook though. It doesn't match any hosts --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 ansible-playbook -v -i "localhost," -c local playbooks/metron_build.yml now it is running and i'll see about the errors --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 Maybe we should add that line to the doc or create a script?? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/436 Well, if we don't want to put deployment things with the src, then ansible is a more flexible and easier to use tool for certain tasks too. But we will talk about it when I get it working --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/436#discussion_r103235131 --- Diff: metron-deployment/roles/kibana/README.md --- @@ -1,35 +0,0 @@ -Kibana 4 --- End diff -- Is this documented anywhere else? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/436#discussion_r103238671 --- Diff: metron-deployment/roles/kibana/README.md --- @@ -1,35 +0,0 @@ -Kibana 4 --- End diff -- The file I commented was the readme that documented how to add or modify the kibana page template --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #436: METRON-671: Refactor existing Ansible de...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/436#discussion_r103239678 --- Diff: metron-deployment/roles/java_jdk/defaults/main.yml --- @@ -15,4 +15,4 @@ # limitations under the License. # --- --- End diff -- Is there a reason not to go to the latest ( _121 ) if we are going to update? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #271: METRON-449 JSONMapParser should unfold maps to ...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/271 Never mind - just found the field name converter --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #271: METRON-449 JSONMapParser should unfold maps to ...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/271 @cestella , from looking at the ES templates etc, it seems that ':' is used as the name separator and not '.'. Should the the unfolding not use the same? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #270: METRON-449 - JSON Parser should unfold nested m...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/270 I don't think I should be carrying the other commits. I didn't rebase. I'll redo --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #270: METRON-449 - JSON Parser should unfold n...
Github user ottobackwards closed the pull request at: https://github.com/apache/incubator-metron/pull/270 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #271: METRON-449 JSONMapParser should unfold m...
GitHub user ottobackwards opened a pull request: https://github.com/apache/incubator-metron/pull/271 METRON-449 JSONMapParser should unfold maps to arbitrary depths The JSON Parser currently only unfolds maps at the root level. The parser would support a wider range of sources is it supported unfolding maps at any depth of the document. METRON-449 adds a recursive function to unfold Maps as they are encountered when parsing. A new function was added because the enum lambdas cannot reference themselves. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/incubator-metron METRON-449 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/271.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #271 commit b5360fff99620d8b6806e1e0fa382b292fa56b8c Author: Otto Fowler <ofow...@industrialdefender.com> Date: 2016-09-24T13:53:31Z METRON-449 JSONMapParser should unfold maps to arbitrary depths --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #284: METRON-474 - fix vagrant ansible default...
GitHub user ottobackwards opened a pull request: https://github.com/apache/incubator-metron/pull/284 METRON-474 - fix vagrant ansible defaults for quick and full Added defaults to match the run.sh parameters and removed the run.sh files. Added a mention in the readmes as well. Now, instead of running run.sh and having to add parameters to vagrant provision, in both quick and full simply running vagrant up and vagrant provision will maintain the correct parameters. Tested by executing vagrant up and provision, vagrant ssh, ambari login You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/incubator-metron METRON-474 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/284.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #284 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #292: METRON-171 add .class to .gitignore
GitHub user ottobackwards opened a pull request: https://github.com/apache/incubator-metron/pull/292 METRON-171 add .class to .gitignore Add .class files to .gitignore You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/incubator-metron METRON-171 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/292.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #292 commit ab74b2cf223787d8d5f6a3c84be88fe815478b34 Author: Otto Fowler <ofow...@industrialdefender.com> Date: 2016-10-05T13:42:04Z METRON-171 add .class to .gitignore --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #293: METRON-473 Add LENGTH() To Stellar
GitHub user ottobackwards opened a pull request: https://github.com/apache/incubator-metron/pull/293 METRON-473 Add LENGTH() To Stellar Add LENGTH to Stellar string functions and unit tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/incubator-metron METRON-473 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/293.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #293 commit 7aaa5d0cd08ae0861601ff2a763f45c71d70dbce Author: Otto Fowler <ofow...@industrialdefender.com> Date: 2016-10-05T14:23:50Z METRON-473 Add LENGTH() function for getting the length of strings to Stellar dsl functions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #284: METRON-474 - fix vagrant ansible default...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/284#discussion_r81776066 --- Diff: metron-deployment/vagrant/quick-dev-platform/README.md --- @@ -5,7 +5,7 @@ This project fully automates the provisioning and deployment of Apache Metron an This image is designed for quick deployment of a single node Metron cluster running on Virtualbox. This platform is ideal for use by Metron developers. It uses a base image that has been pre-loaded with Ambari and HDP. -Metron is composed of many components and installing all of these on a single host, especially a virtualized one, will greatly stress the resources of the host. The host will require at least 8 GB of RAM and a fair amount of patience. It is highly recommended that you shut down all unnecessary services. +Metron is composed of many components and installing all of these on a single host, especially a virtualized one, will greatly stress the resources of the host. The host will require at least 8 GB of RAM and a fair amount of patience. It is highly recommended that you shut down all unnecessary services. To that end the vagrant file configuration defaults to disabling solr and yaf. --- End diff -- Done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #293: METRON-473 Add LENGTH() To Stellar
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/293 @mmiklavc I had seen that. The tests in StellarTests test the compiler and the variable resolution etc as well is my point. It is worth having both. If both were done from the start, maybe METRON-439 doesn't get through --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #296: METRON-439: Stellar : IS_EMPTY(host) throws exc...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/296 OK, if I change my functions to not throw then it works. This doesn't seem right to me, but it may be something to do with why the tests are run that way. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #293: METRON-473 Add LENGTH() To Stellar
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/293 OK I have it in. Here is the thing that I found, I know there is a pull about fixing IS_EMPTY from throwing exceptions - and I had to take that same return 0 approach instead of throwing. IS_EMPTY did NOT have a test in StellarTests, so it was never being tested through the stellar variable resolver, validation, parser etc. If it had had a test there ( using the run() as implemented in the test ) it never would have been committed, because it NEVER works, no matter what you pass it as originally implemented with the IllegalStateExceptions. This is because the validate calls down to the apply with a specifically null argument which triggers the exception. So, a couple of things: - every stellar function needs to be tested through the whole stellar framework and not just on top of StellarFunction.apply() - why would validate call parse which calls apply? Is that the way it is at runtime? Does apply always get called twice? Is the test harness wrong? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #296: METRON-439: Stellar : IS_EMPTY(host) throws exc...
Github user ottobackwards commented on the issue:
https://github.com/apache/incubator-metron/pull/296
I'm working on making LENGTH work like IS_EMPTY. I can't get either one to
work as both are failing validation. There is no StellarTest for IS_EMPTY ( I
have added one to check if it is just my code that is failing ). Can you one
and try it there an make sure it works all the way through the Stellar parse
and variable resolve?
ParseException: Unable to parse IS_EMPTY(foo): Unable to execute: IS_EMPTY
expects a collection or a string
This is when calling with
`String query = "IS_EMPTY(foo);
Asser.assertEquals(true,run(query,ImmutableMap.of("foo",""));
`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #293: METRON-473 Add LENGTH() To Stellar
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/293#discussion_r82259618 --- Diff: metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java --- @@ -30,9 +30,7 @@ import org.reflections.Reflections; import org.reflections.util.ConfigurationBuilder; -import java.util.HashMap; -import java.util.HashSet; -import java.util.Map; +import java.util.*; --- End diff -- I did not actually change this on purpose, I'll fix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #293: METRON-473 Add LENGTH() To Stellar
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/293 I took this jira to get some start of an idea about Stellar, and after debugging through it to track down to exitVariable to find the argument resolution and then back up to the validate x->null I would say I got what I bargained for. I am not sure how I would answer your question on validation, I don't know Stellar well enough. My experience and intuition tells me that running the 'execute' and using an exception or error case as validation is not very efficient at runtime, if in fact this is what happens at runtime as well. Better to have real metadata on the function and just validating the metadata and they syntax of the query I would think. *A logical/runtime error does not mean an invalid expression*. This method is in effect equating them which I think is incorrect ( unless I am mistaking the intent of validate()). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #293: METRON-473 Add LENGTH() To Stellar
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/293 It would make a nice little POC - --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #293: METRON-473 Add LENGTH() To Stellar
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/293 ok, as above, I'm not sure about x->null to do that, but I am the noob here --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #293: METRON-473 Add LENGTH() To Stellar
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/293 @cestella I can do that. Should I move LENGTH to DataStructureFunctions then? Also - I notice that IS_EMPTY throws IllegalStateExceptions instead of returning null, is that how no-var should be handled? In the String functions they return null --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #299: METRON-425 Stellar transformation fails ...
GitHub user ottobackwards opened a pull request: https://github.com/apache/incubator-metron/pull/299 METRON-425 Stellar transformation fails to handle special characters The issue is that per our grammar the arithmetic and comparison operators need to be escaped to be used as literals. I have added a unit test to the system for this case ( for field transformations ). I have also taken a stab at updating the readme.md. I do not thing the readme is correct though. It may need to be written out more better by someone who writes good. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/incubator-metron METRON-425 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/299.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #299 commit ac2ffc4d4c5c9f233e3629b5884b0e0073646823 Author: Otto Fowler <ofow...@industrialdefender.com> Date: 2016-10-06T22:31:04Z METRON-425 add unit test and readme change about escaping literals in stellar statements --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #299: METRON-425 Stellar transformation fails to hand...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/299 So - based on `identifier_operand : (TRUE | FALSE) # LogicalConst | arithmetic_expr #ArithmeticOperands | STRING_LITERAL # StringLiteral | list_entity #List | map_entity #MapConst | NULL #NullConst | EXISTS LPAREN IDENTIFIER RPAREN #ExistsFunc | LPAREN conditional_expr RPAREN#condExpr_paren ;` Anything that can be completely expressed from this list will work. TRUE and FALSE can, but NOT um, can not. It is not random. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #293: METRON-473 Add LENGTH() To Stellar
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/293#discussion_r82218396
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java
---
@@ -222,4 +222,16 @@ public Object apply(List args) {
return null;
}
}
+
+ @Stellar( name="LENGTH"
+ , description = "Returns the length of a string"
+ , params = { "input - String" }
+ , returns = "String"
--- End diff --
No - I need to fix that. I started with the TO_UPPER def. to make sure I
got it right and need to change that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #293: METRON-473 Add LENGTH() To Stellar
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/293#discussion_r82218813
--- Diff:
metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
---
@@ -285,6 +285,18 @@ public void testHappyPath() {
}
@Test
+ public void testLength(){
+String query = "LENGTH(foo)";
+Assert.assertEquals(5, run(query,ImmutableMap.of("foo","abcde")));
+ }
+
+ @Test
--- End diff --
I will fix - see my comment above about how to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #293: METRON-473 Add LENGTH() To Stellar
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/293#discussion_r82218691
--- Diff:
metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
---
@@ -285,6 +285,18 @@ public void testHappyPath() {
}
@Test
--- End diff --
So, I was going to make a test passing in NULL, but the Immutable map will
not take nulls, can you clarify how you would test this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #261: METRON-434: JSON Parser
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/261 Casey, would you feel about adding in recursion, such that nested maps will be unfolded? I have that working. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #261: METRON-434: JSON Parser
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/261 No problem - do you require a jira? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #261: METRON-434: JSON Parser
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/261 Should I open a new jira and do a PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #261: METRON-434: JSON Parser
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/261 METRON-449 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #261: METRON-434: JSON Parser
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/261 Is it correct that multiple separate instances of this parser ( or any parser ) could be configured by adding multiple zookeeper jsons with separate kafka topics? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #279: METRON-466: Full Dev Platform build some...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/279#discussion_r80999009 --- Diff: metron-deployment/inventory/quick-dev-platform/group_vars/all --- @@ -0,0 +1,95 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# which services should be started? +services_to_start: + - mysql + - elasticsearch + - pcap-service + - kibana + - snort + - snort-logs + - bro + - pcap-replay + - bro-parser + - snort-parser + - enrichment + - indexing --- End diff -- For what it's worth that is a good idea. Maybe there could be some 'special' scripts or tools available for that configuration ( things you would want while developing a parser and stellar rules etc ) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #283: METRON-472: add runAgain.sh to handle a ...
GitHub user ottobackwards opened a pull request: https://github.com/apache/incubator-metron/pull/283 METRON-472: add runAgain.sh to handle a proper vagrant provision When you fail after running run.sh in quickdev you will be told to run vagrant provision... but what you really want to do is run vagrant provision with the same parameters that you used for vagrant up inside run.sh. runAgain.sh has that. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/incubator-metron METRON-472 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/283.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #283 commit 820045a68bf8ff7c020fbe3986cbbe45cdaf803b Author: Otto Fowler <ofow...@industrialdefender.com> Date: 2016-09-29T04:26:22Z METRON-472: add runAgain.sh to handle a proper vagrant provision after a run.sh failure --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #283: METRON-472: add runAgain.sh to handle a proper ...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/283 No you are right I thought I closed the pr, I guess I did not. Sorry --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #325: METRON-512 up default junit to 4.12
GitHub user ottobackwards opened a pull request: https://github.com/apache/incubator-metron/pull/325 METRON-512 up default junit to 4.12 "Global default JUnit version is 4.4. It would be nice to update this to 4.12 and expose newer JUnit featues. Existing pom files should be updated - some are pointing to the version in an ad-hoc fashion rather than using the global version." full build and tests passing You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/incubator-metron METRON-512 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/325.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #325 commit cae270288c32824fc256d5155752e792e7fd16ff Author: Otto Fowler <ofow...@industrialdefender.com> Date: 2016-10-26T02:54:51Z METRON-512 up default junit t0 4.12, cleanup references to use global --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #329: METRON-148 compress logs with logrotate
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/329 kicking travis --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #329: METRON-148 compress logs with logrotate
Github user ottobackwards closed the pull request at: https://github.com/apache/incubator-metron/pull/329 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #329: METRON-148 compress logs with logrotate
GitHub user ottobackwards reopened a pull request: https://github.com/apache/incubator-metron/pull/329 METRON-148 compress logs with logrotate Add compress configuration for logrotate.d/ configurations Tested using the logrotate -f /etc/logrotate.d/metron-* commands You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/incubator-metron METRON-148 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/329.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #329 commit b44146cfceb542fbeed4e812a8fb9a62fa29bfbe Author: Otto Fowler <ofow...@industrialdefender.com> Date: 2016-10-27T01:50:13Z METRON-148 compress logs with logrotate --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/318 I think this monit timeout issue is part of the problem on low resource machines, and with 'zombie' storm threads being left behind --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #318: METRON-495: Upgrade Storm to 1.0.x
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/318 Are there instructions for doing either of those things? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #330: Metron 506 - Add Otto Fowler to committers
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/330 Kicking travis --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #330: Metron 506 - Add Otto Fowler to committe...
GitHub user ottobackwards opened a pull request: https://github.com/apache/incubator-metron/pull/330 Metron 506 - Add Otto Fowler to committers Add Otto Fowler to committers list on main site You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/incubator-metron METRON-506 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/330.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #330 commit 4c0426a7ecc1d0e26837b8016f3432ded0009783 Author: Otto Fowler <ofow...@industrialdefender.com> Date: 2016-10-17T19:54:23Z added Otto Fowler to commiters commit 401ece493ffc753cc8c35afcb945f6cd9eec9a66 Author: Otto Fowler <ofow...@industrialdefender.com> Date: 2016-10-22T02:45:17Z Merge branch 'master' of https://github.com/apache/incubator-metron into METRON-506 commit c98d11cc0cc0594f4465fd5cc4cf1358f0cff3a0 Author: Otto Fowler <ofow...@industrialdefender.com> Date: 2016-10-28T16:00:12Z merge remote tracking branch apache master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #330: Metron 506 - Add Otto Fowler to committe...
GitHub user ottobackwards reopened a pull request: https://github.com/apache/incubator-metron/pull/330 Metron 506 - Add Otto Fowler to committers Add Otto Fowler to committers list on main site You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/incubator-metron METRON-506 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/330.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #330 commit 4c0426a7ecc1d0e26837b8016f3432ded0009783 Author: Otto Fowler <ofow...@industrialdefender.com> Date: 2016-10-17T19:54:23Z added Otto Fowler to commiters commit 401ece493ffc753cc8c35afcb945f6cd9eec9a66 Author: Otto Fowler <ofow...@industrialdefender.com> Date: 2016-10-22T02:45:17Z Merge branch 'master' of https://github.com/apache/incubator-metron into METRON-506 commit c98d11cc0cc0594f4465fd5cc4cf1358f0cff3a0 Author: Otto Fowler <ofow...@industrialdefender.com> Date: 2016-10-28T16:00:12Z merge remote tracking branch apache master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
