[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack

2018-03-15 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/946
  
thanks @mmiklavc 


---


[GitHub] metron issue #964: METRON-1491: The indexing topology restart logic is wrong

2018-03-15 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/964
  
Yes, lets get it in there


---


[GitHub] metron issue #964: METRON-1491: The indexing topology restart logic is wrong

2018-03-15 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/964
  
Should we think about exposing them as separate things in ambari over all?  
Go all the way with this?


---


[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack

2018-03-09 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/946
  
@cestella " Yeah, I think that's the approach, however, there's a snag. 
Storm requires us to create uber jars, so probably what we want to do is have 
users actually put the xpath transport client on the storm.library.path."  If 
only there was a way to load things into storm with classloader isolation and 
dependency inclusion.


---


[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack

2018-03-09 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/946
  
would this have any effect on people using x-pack alternatives?


---


[GitHub] metron issue #946: METRON-1465:Support for Elasticsearch X-pack

2018-03-09 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/946
  
wait, does this PR mean we *require* x-pack from now on?


---


[GitHub] metron issue #951: Metron-1479 Add editorconfig - create guideline for code ...

2018-03-07 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/951
  
Why are there so many changes to the package locks vs. to the package files 
themselves?


---


[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

2018-03-06 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/943
  
What about upgrading.md?  


---


[GitHub] metron issue #942: METRON-1461: Modify the MIN, MAX Stellar methods to take ...

2018-03-06 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/942
  
Maybe making this function so generic is going to necessitate it being so 
complicated that it is harder to maintain etc.


---


[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...

2018-03-05 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/940
  
Maybe the issue has to do with our keys, and their distribution as the size 
get's larger?  Maybe when we get larger sizes we get more collisions and end up 
calling equals() more or something.


---


[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...

2018-03-05 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/940
  
This should have the equiv. diagram and documentation ( i believe as shown 
above ) to the original split join strategy.



---


[GitHub] metron-bro-plugin-kafka pull request #6: Configurable JSON timestamps and de...

2018-03-05 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:


https://github.com/apache/metron-bro-plugin-kafka/pull/6#discussion_r172243410
  
--- Diff: src/KafkaWriter.cc ---
@@ -54,20 +66,49 @@ KafkaWriter::KafkaWriter(WriterFrontend* frontend): 
WriterBackend(frontend), for
 }
 
 KafkaWriter::~KafkaWriter()
-{}
+{
+
+// Cleanup all the things
+delete topic;
+delete producer;
+delete formatter;
+delete conf;
+delete topic_conf;
+
+}
 
 bool KafkaWriter::DoInit(const WriterInfo& info, int num_fields, const 
threading::Field* const* fields)
 {
+// Timeformat object, default to TS_EPOCH
+threading::formatter::JSON::TimeFormat tf = 
threading::formatter::JSON::TS_EPOCH;
+
 // if no global 'topic_name' is defined, use the log stream's 'path'
 if(topic_name.empty()) {
 topic_name = info.path;
 }
 
+// format timestamps
+if ( strcmp(json_timestamps.c_str(), "JSON::TS_EPOCH") == 0 ) {
--- End diff --

Maybe we should put an implementation comment explaining that in there


---


[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...

2018-03-02 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/940
  
have we thought to send a mail to the storm dev list and ask if anyone has 
done this?  potential issues?


---


[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...

2018-03-02 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/940
  
If we integrated storm with yarn this would also be a problem, as our 
resource management may be at odds with yarn's.  I think?

What would be nice is if storm could manage the pool and we could just use 
it.


---


[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

2018-03-01 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/853
  
+1 by inspection


---


[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

2018-03-01 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/853
  
Sorry, I'll try to get back to this today



---


[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

2018-02-23 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/943
  
I think I mentioned contrib.  They don't have to go in contrib, I think at 
the time someone mentioned not wanting to maintain them.. If we don't then I 
thought contrib would make sense.
We can keep them normal.

Metron installation artifacts ( templates, schemas, kibana objects, 
queries, and dashboards ) should be managed separate from the kibana / es / 
solr /  packs.
They are artifacts of metron and we need to support installing them on 
existing installations.
If they need to be installed in a different phase, or as their own 
component, then fine.
we need to break that down more anyway ( like package the 'demo' system + 
parser configs as optional ) maybe.






---


[GitHub] metron pull request #853: METRON-1337: List of facets should not be hardcode...

2018-02-22 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/853#discussion_r170125479
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/AlertServiceImpl.java
 ---
@@ -37,15 +47,21 @@
 @Service
 public class AlertServiceImpl implements AlertService {
 
--- End diff --

I think that is great.  At some point, if we are going to do this a few 
times we should have a standard naming convention ( separators at least or 
something).  But that isn't a thing for this PR


---


[GitHub] metron pull request #940: Single bolt split join poc

2018-02-22 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/940#discussion_r170056794
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java
 ---
@@ -0,0 +1,323 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.enrichment.bolt;
+
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.bolt.ConfiguredEnrichmentBolt;
+import org.apache.metron.common.configuration.ConfigurationType;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.handler.ConfigHandler;
+import org.apache.metron.common.error.MetronError;
+import org.apache.metron.common.performance.PerformanceLogger;
+import org.apache.metron.common.utils.ErrorUtils;
+import org.apache.metron.common.utils.MessageUtils;
+import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase;
+import org.apache.metron.enrichment.configuration.Enrichment;
+import org.apache.metron.enrichment.interfaces.EnrichmentAdapter;
+import org.apache.metron.enrichment.parallel.EnrichmentContext;
+import org.apache.metron.enrichment.parallel.EnrichmentStrategies;
+import org.apache.metron.enrichment.parallel.ParallelEnricher;
+import org.apache.metron.enrichment.parallel.WorkerPoolStrategy;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.stellar.dsl.StellarFunction;
+import org.apache.metron.stellar.dsl.StellarFunctions;
+import org.apache.storm.task.OutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Tuple;
+import org.apache.storm.tuple.Values;
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.UnsupportedEncodingException;
+import java.lang.invoke.MethodHandles;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+
--- End diff --

This needs some doc, comments - something in hear to help with review and 
maintenance of what is going on in here.


---


[GitHub] metron pull request #853: METRON-1337: List of facets should not be hardcode...

2018-02-20 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/853#discussion_r169515112
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/AlertServiceImpl.java
 ---
@@ -37,15 +47,21 @@
 @Service
 public class AlertServiceImpl implements AlertService {
 
--- End diff --

That seems reasonable


---


[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

2018-02-20 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/853
  
This looks great, one small comment from review


---


[GitHub] metron pull request #853: METRON-1337: List of facets should not be hardcode...

2018-02-20 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/853#discussion_r169441455
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/AlertServiceImpl.java
 ---
@@ -37,15 +47,21 @@
 @Service
 public class AlertServiceImpl implements AlertService {
 
--- End diff --

can we make the type metron_alert?  alert seems a little generic


---


[GitHub] metron issue #939: [BUG-96727] xpack support (for discussion only)

2018-02-19 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/939
  
It is almost like hw has a different issue tracking system


---


[GitHub] metron issue #939: [BUG-96727] xpack support (for discussion only)

2018-02-19 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/939
  
BUG- is not a valid title for this project. 


---


[GitHub] metron issue #939: [BUG-96727] xpack support (for discussion only)

2018-02-19 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/939
  
Can you put some kind of description in here?  What does that BUG refer to?


---


[GitHub] metron issue #579: METRON-941 native PaloAlto parser corrupts message when h...

2018-02-16 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/579
  
+1 one.  Thank you @ctramnitz.  I know how it can be to have a long review 
process.  This is really good work and I look forward to seeing what you do 
next ;)


---


[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

2018-02-16 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/853
  
I am not an expert in hbase, I cannot say how i'd implement it.
The version thing we can leave out, it will be just as good or bad as what 
we have in zookeeper right?

I think the use case is:

For any given User ( at least ) we need the ability to store currently 
known and future unknown configurations as bytes, with configuration type 
identifier lookup.  This should be done such that new configuration types can 
be added by 'putting' config bytes with a new identifier.

I think that gets us a long way.


---


[GitHub] metron pull request #579: METRON-941 native PaloAlto parser corrupts message...

2018-02-16 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/579#discussion_r168779259
  
--- Diff: Upgrading.md ---
@@ -19,6 +19,24 @@ limitations under the License.
 This document constitutes a per-version listing of changes of
 configuration which are non-backwards compatible.
 
+## 0.4.2 to 0.4.3
+
+### [METRON-941: native PaloAlto parser corrupts message when having a 
comma in the payload](https://issues.apache.org/jira/browse/METRON-941)
+While modifying the PaloAlto log parser to support logs from newer
+PAN-OS version and to not break when a message payload contains a
--- End diff --

change/changed


---


[GitHub] metron issue #579: METRON-941 native PaloAlto parser corrupts message when h...

2018-02-16 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/579
  
maybe regression isn't the right word  - 
"Until we have something generic to (pre-)parse syslog before it reaches 
the message parser I assumed the messages will be stripped off the syslog 
header for now." is this not a new requirement or behavior?


---


[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

2018-02-16 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/853
  
Of course re-reading your comment, I see you specifically didn't account 
for these, but I think there are some basic tweaks to allow for this that don't 
presuppose far flung 20% use cases.


---


[GitHub] metron issue #853: METRON-1337: List of facets should not be hardcoded

2018-02-16 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/853
  
First, nice work.
Second, I have this question:

Since we will have some unknown number of rest clients, that may want to 
save 'configuration' associated with a user, and not just Alerts UI, should it 
not be factored more generically?

getUserSettingsForType()

And stored as user, type, settings byte[]  or some such?

We may also want to store the version so that it is user, type, 
settingsFmtVersion, settings.

We might also want to consider storing as Protobufs, which are more 
supporting changes.




---


[GitHub] metron issue #579: METRON-941 fix PaloAltoParser

2018-02-16 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/579
  
after that I'll merge



---


[GitHub] metron issue #579: METRON-941 fix PaloAltoParser

2018-02-16 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/579
  
@ctramnitz one thing, since this is a regression, technically, we need to 
update the release notes / upgrade guide.
Can you add a note to the Upgrading.md about the removal of the Syslog 
-ness?


---


[GitHub] metron issue #579: METRON-941 fix PaloAltoParser

2018-02-13 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/579
  
Im +1 on this.  I would like to get comment from @simonellistonball et al 
on the change for syslog


---


[GitHub] metron issue #934: METRON-1423: Ambari work to handle Solr configuration

2018-02-12 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/934
  
I would just like it documented as to why we are not using the HDP solr 
mpack.
Although, I would think we would be using that mpack as the example for 
ours?


---


[GitHub] metron issue #934: METRON-1423: Ambari work to handle Solr configuration

2018-02-11 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/934
  
Can I ask a question : why aren't we using 
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_solr-search-installation/content/ch_hdp-search-install-ambari.html?
Is there a reason?  If so we should document that reason.



---


[GitHub] metron issue #934: METRON-1423: Ambari work to handle Solr configuration

2018-02-11 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/934
  
Ran through test, everything worked fine. +1

On the questions:

- What Solr parameters do we expose in Ambari? All of them?

People are going to want to tune indexing.  If ambari is managing the 
configuration where those tuning parameters happen, then they need to be exposed

- Do we want to add Solr collection functions to Service actions? 

Does that mean that we can stop a certain collection?  We may want to have 
a custom view for this?





---


[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...

2018-02-08 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/929
  
+1


---


[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...

2018-02-08 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/929
  
@merrimanr did you push the configs to zookeeper?



---


[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...

2018-02-08 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/929
  
Everything worked, just wondering about the questions above before I check 
off.



---


[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...

2018-02-08 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/929
  
don't we have to push the new config to zookeeper?


---


[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...

2018-02-08 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/929
  
Why is it "solr.zookeeper" : "localhost:9983"  and not solr.ip?



---


[GitHub] metron issue #914: METRON-1397 Support for JSON Path and complex documents i...

2018-02-08 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/914
  
Um, it is updated.


---


[GitHub] metron issue #914: METRON-1397 Support for JSON Path and complex documents i...

2018-02-07 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/914
  
Just a note on parser docs, part of the mythical 777 is support for per 
parser readme, that would be nice here.


---


[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...

2018-02-07 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/929
  
Cool, I'll run the test plan


---


[GitHub] metron issue #914: METRON-1397 Support for JSON Path and complex documents i...

2018-02-07 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/914
  
yeah, you said in the docs, will do


---


[GitHub] metron issue #914: METRON-1397 Support for JSON Path and complex documents i...

2018-02-07 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/914
  
http://goessner.net/articles/JsonPath/
https://github.com/json-path/JsonPath
http://jsonpath.herokuapp.com


https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-testing.html


---


[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...

2018-02-07 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/929
  
See, we explored it.  Thanks!!


---


[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...

2018-02-07 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/929
  
Sorry, maybe I misunderstood what I have read here: 

https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

I thought that you could set to not have to manually commit.



---


[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...

2018-02-07 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/929
  
Should we explore auto-commit?
Also, maybe we should rename all this stuff to SolrCloud, since Solr and 
SolrCloud *are* different in some use of terms and concepts?


---


[GitHub] metron issue #928: METRON-1444: Add Ubuntu Repositories for Elasticsearch to...

2018-02-07 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/928
  
I'll add 2 smileys next time, to help get my point across.  I would have 
expected a response more along the lines of:
"Upgrading Ambari to the 2.6 release line shows little risk when I looked 
at it, as Ambari seems to be fairly important to Metron."


---


[GitHub] metron issue #928: METRON-1444: Add Ubuntu Repositories for Elasticsearch to...

2018-02-07 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/928
  
Shouldn't the title of this PR be "Update ambari from 2.5.x to 2.6.x"?  
That seems the bigger thing :)


---


[GitHub] metron issue #922: METRON-1441: Create complementary Solr schemas for the ma...

2018-02-05 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/922
  
So, the idea is that we have a 'core' for each parser?  and for errors?


---


[GitHub] metron pull request #924: METRON-1299 In MetronError tests, don't test for H...

2018-02-02 Thread ottobackwards
GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/924

METRON-1299 In MetronError tests, don't test for HostName if getHostName 
wouldn't work

MetronError ignores exceptions from 
InetAddress.getLocalHost().getHostName() and leaves the field unset.

The unit test however assumes it would be set, and someone has logged a 
jira on this, since it makes the build fail.

Changed the test so that it only verifies the hostName if it would have 
worked.

### Testing
- Code review
- Tests Pass

> no non-test changes in pr

```java
 private void addHostname(JSONObject errorMessage) {
try {
  errorMessage.put(ErrorFields.HOSTNAME.getName(), 
InetAddress.getLocalHost().getHostName());
} catch (UnknownHostException ex) {
  // Leave the hostname field off if it cannot be found
}
  }
```

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron error_addHost

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/924.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #924


commit cee3acba914f97ca7d2faf6e7822c97928a4e242
Author: Otto Fowler <ottobackwards@...>
Date:   2018-02-02T21:57:47Z

do not test for hostName if calling hostName throws, since it will be null




---


[GitHub] metron issue #865: METRON-1212 The bundle System and Maven Plugin (Feature B...

2018-02-02 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/865
  
Hey @JonZeolla how is it going?



---


[GitHub] metron pull request #922: METRON-1441: Create complementary Solr schemas for...

2018-02-02 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/922#discussion_r165662614
  
--- Diff: 
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/schema/SchemaTranslatorTest.java
 ---
@@ -0,0 +1,188 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.schema;
+
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.integration.UnableToStartException;
+import org.apache.metron.solr.integration.components.SolrComponent;
+import org.apache.metron.solr.writer.SolrWriter;
+import org.json.simple.JSONObject;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.util.*;
+
+public class SchemaTranslatorTest {
+
+  /**

+{"adapter.threatinteladapter.end.ts":"1517499201357","bro_timestamp":"1517499194.7338","ip_dst_port":8080,"enrichmentsplitterbolt.splitter.end.ts":"1517499201202","enrichmentsplitterbolt.splitter.begin.ts":"1517499201200","adapter.hostfromjsonlistadapter.end.ts":"1517499201207","adapter.geoadapter.begin.ts":"1517499201209","uid":"CUrRne3iLIxXavQtci","trans_depth":143,"protocol":"http","original_string":"HTTP
 | id.orig_p:50451 method:GET request_body_len:0 id.resp_p:8080 
uri:\/api\/v1\/clusters\/metron_cluster\/services\/KAFKA\/components\/KAFKA_BROKER?fields=metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsBytesInPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsBytesOutPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsMessagesInPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/controller\/KafkaController\/ActiveControllerCount[1484165330,1484168930,15],metrics\/kafk
 
a\/controller\/ControllerStats\/LeaderElectionRateAndTimeMs\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/controller\/ControllerStats\/UncleanLeaderElectionsPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaFetcherManager\/Replica-MaxLag[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaManager\/PartitionCount[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaManager\/UnderReplicatedPartitions[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaManager\/LeaderCount[1484165330,1484168930,15]=null_padding&_=1484168930776
 tags:[] uid:CUrRne3iLIxXavQtci referrer:http:\/\/node1:8080\/ trans_depth:143 
host:node1 id.orig_h:192.168.66.1 response_body_len:0 user_agent:Mozilla\/5.0 
(Macintosh; Intel Mac OS X 10_12_2) AppleWebKit\/537.36 (KHTML, like Gecko) 
Chrome\/55.0.2883.95 Safari\/537.36 ts:1517499194.7338 
id.resp_h:192.168.66.121","ip_dst_addr":"192.168.66.121","threatinteljoinbolt.joiner.ts":"1517499201359","host":"node1","en
 
richmentjoinbolt.joiner.ts":"1517499201212","adapter.hostfromjsonlistadapter.begin.ts":"1517499201206","threatintelsplitterbolt.splitter.begin.ts":"1517499201215","ip_src_addr":"192.168.66.1","user_agent":"Mozilla\/5.0
 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit\/537.36 (KHTML, like Gecko) 
Chrome\/55.0.2883.95 
Safari\/537.36","timestamp":1517499194733,"method":"GET","request_body_len":0,"uri":"\/api\/v1\/clusters\/metron_cluster\/services\/KAFKA\/components\/KAFKA_BROKER?fields=metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsBytesInPerSec\/1Mi

[GitHub] metron pull request #920: METRON-1438 Move SHELL functions from metron-manag...

2018-02-02 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165662017
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/ShellFunctionsTest.java
 ---
@@ -40,8 +45,8 @@
   );
 
   Context context = new Context.Builder()
-.with(Context.Capabilities.SHELL_VARIABLES , () -> variables)
-.build();
+.with(Context.Capabilities.SHELL_VARIABLES , () -> 
variables).build();
--- End diff --

How do you have your formatting preferences set to get the above?


---


[GitHub] metron pull request #920: METRON-1438 Move SHELL functions from metron-manag...

2018-02-02 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165661676
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/ShellFunctionsTest.java
 ---
@@ -40,8 +45,8 @@
   );
 
   Context context = new Context.Builder()
-.with(Context.Capabilities.SHELL_VARIABLES , () -> variables)
-.build();
+.with(Context.Capabilities.SHELL_VARIABLES , () -> 
variables).build();
--- End diff --

I didn't reformat that like that on purpose.  I had a period where I was 
setting the CONSOLE capability.  When I removed it, it just worked out like 
this.

Even if I select and format in intellij it doesn't change the .build() to 
the next line.

I think what you have above is fine.  I would like it to 'just' happen if I 
format code though, since it is easy for things to slip through.

It is tough right now, because so much of the codebase isn't formatted to 
check style, and I don't think we want every pr to include a lot of formatting 
changes.



---


[GitHub] metron pull request #920: METRON-1438 Move SHELL functions from metron-manag...

2018-02-02 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165659854
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/cli/PausableInput.java
 ---
@@ -36,8 +37,8 @@
  *
  */
 public class PausableInput extends InputStream {
-  InputStream in = System.in;
-  boolean paused = false;
+  private InputStream in = System.in;
+  private AtomicBoolean paused = new AtomicBoolean(false);
--- End diff --

I found in travis and locally, that the PausableInput was hanging, my 
builds where not completing.  PausableInput.INSTANCE.unpause(); was never 
returning.

I couldn't figure out why it was locked.  Reading the class documentation 
and the trying to understand the threading in the shell, I looked at the class 
and saw some inconsistencies with how and when we checked if we where paused in 
the different read() calls, and also that the flag field was not volatile or 
atomic.  I refactored this things to what I *think* would be the correct and 
consistent approaches and  it resolved my issue.


---


[GitHub] metron pull request #922: METRON-1441: Create complementary Solr schemas for...

2018-02-01 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/922#discussion_r165548937
  
--- Diff: 
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/schema/SchemaTranslatorTest.java
 ---
@@ -0,0 +1,188 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.schema;
+
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.integration.UnableToStartException;
+import org.apache.metron.solr.integration.components.SolrComponent;
+import org.apache.metron.solr.writer.SolrWriter;
+import org.json.simple.JSONObject;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.util.*;
+
+public class SchemaTranslatorTest {
+
+  /**

+{"adapter.threatinteladapter.end.ts":"1517499201357","bro_timestamp":"1517499194.7338","ip_dst_port":8080,"enrichmentsplitterbolt.splitter.end.ts":"1517499201202","enrichmentsplitterbolt.splitter.begin.ts":"1517499201200","adapter.hostfromjsonlistadapter.end.ts":"1517499201207","adapter.geoadapter.begin.ts":"1517499201209","uid":"CUrRne3iLIxXavQtci","trans_depth":143,"protocol":"http","original_string":"HTTP
 | id.orig_p:50451 method:GET request_body_len:0 id.resp_p:8080 
uri:\/api\/v1\/clusters\/metron_cluster\/services\/KAFKA\/components\/KAFKA_BROKER?fields=metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsBytesInPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsBytesOutPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsMessagesInPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/controller\/KafkaController\/ActiveControllerCount[1484165330,1484168930,15],metrics\/kafk
 
a\/controller\/ControllerStats\/LeaderElectionRateAndTimeMs\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/controller\/ControllerStats\/UncleanLeaderElectionsPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaFetcherManager\/Replica-MaxLag[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaManager\/PartitionCount[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaManager\/UnderReplicatedPartitions[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaManager\/LeaderCount[1484165330,1484168930,15]=null_padding&_=1484168930776
 tags:[] uid:CUrRne3iLIxXavQtci referrer:http:\/\/node1:8080\/ trans_depth:143 
host:node1 id.orig_h:192.168.66.1 response_body_len:0 user_agent:Mozilla\/5.0 
(Macintosh; Intel Mac OS X 10_12_2) AppleWebKit\/537.36 (KHTML, like Gecko) 
Chrome\/55.0.2883.95 Safari\/537.36 ts:1517499194.7338 
id.resp_h:192.168.66.121","ip_dst_addr":"192.168.66.121","threatinteljoinbolt.joiner.ts":"1517499201359","host":"node1","en
 
richmentjoinbolt.joiner.ts":"1517499201212","adapter.hostfromjsonlistadapter.begin.ts":"1517499201206","threatintelsplitterbolt.splitter.begin.ts":"1517499201215","ip_src_addr":"192.168.66.1","user_agent":"Mozilla\/5.0
 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit\/537.36 (KHTML, like Gecko) 
Chrome\/55.0.2883.95 
Safari\/537.36","timestamp":1517499194733,"method":"GET","request_body_len":0,"uri":"\/api\/v1\/clusters\/metron_cluster\/services\/KAFKA\/components\/KAFKA_BROKER?fields=metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsBytesInPerSec\/1Mi

[GitHub] metron pull request #922: METRON-1441: Create complementary Solr schemas for...

2018-02-01 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/922#discussion_r165548986
  
--- Diff: 
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/schema/SchemaTranslatorTest.java
 ---
@@ -0,0 +1,188 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.schema;
+
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.integration.UnableToStartException;
+import org.apache.metron.solr.integration.components.SolrComponent;
+import org.apache.metron.solr.writer.SolrWriter;
+import org.json.simple.JSONObject;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.util.*;
+
--- End diff --

Shouldn't this be an integration test?


---


[GitHub] metron pull request #920: METRON-1438 Move SHELL functions from metron-manag...

2018-02-01 Thread ottobackwards
GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/920

METRON-1438 Move SHELL functions from metron-management to stellar-common

Part of making stellar more modular and stand alone is consolidation of 
stellar functionality scattered through metron into stellar common.   That 
should be done if the functionality is generally useful and not metron 
specific. 

Where the functionality *is* metron specific, we should look to factor such 
functionality as to extract generally useful functionality.

In the case of the SHELL- namespace, it is clearly not metron specific.

This pr moves this functionality from the metron-management module to 
stellar common.

## Testing
- build and tests should run as normal
- shell functions such as those in the metron-management readme examples 
should run
- functions should work as before


### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [na] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [na] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [na] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron stellar-move-shell-funcs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/920.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #920


commit fe93fc40d94cc6148667eccc36853a2dc9925546
Author: Otto Fowler <ottobackwards@...>
Date:   2018-02-01T14:28:46Z

move ShellFunctions to stellar-common

commit 38775033240c7c174c99b10ffb5087c39b23672a
Author: Otto Fowler <ottobackwards@...>
Date:   2018-02-01T15:33:42Z

cleanup and refactor for failed tests where PausableInput did not unpause

commit 7af3d347567fe967c9c2ea1191c121a470d2bf34
Author: Otto Fowler <ottobackwards@...>
Date:   2018-02-01T18:28:43Z

fix readme




---


[GitHub] metron issue #919: METRON-1439: Turn off git pager in platform-info script

2018-02-01 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/919
  
No, I am sorry, I thought that was understood.  I have not reviewed, but I 
would go ahead based on @cestella 


---


[GitHub] metron issue #919: METRON-1439: Turn off git pager in platform-info script

2018-02-01 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/919
  
this is a user utility though, not a dev one.  I would create a new 
support-scripts folder


---


[GitHub] metron issue #579: METRON-941 fix PaloAltoParser

2018-02-01 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/579
  
@ctramnitz thank you!  Let us know where you are at and if we can help


---


[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts

2018-01-31 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/857
  
What is the status of this pr?  it is 29 day without comment, and 
conflicted, literally, and perhaps figuratively



---


[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

2018-01-31 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r165048687
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
 ---
@@ -0,0 +1,106 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static 
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+  private static final String FAILED_COMPILE = "Failed to compile";
+  private CuratorFramework client;
+
+  public StellarZookeeperBasedValidator(CuratorFramework client) throws 
NullArgumentException {
+if (client == null) {
+  throw new NullArgumentException("client");
+}
+this.client = client;
+  }
+
+
+  @Override
+  public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c : 
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --

@cestella ping


---


[GitHub] metron issue #873: METRON-1367 Stellar should have some instrumentation of f...

2018-01-30 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/873
  
bump



---


[GitHub] metron issue #690: METRON-1091 Package STELLAR shell as stand alone

2018-01-30 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/690
  
Done, let me know if the changes are sufficient @JonZeolla 


---


[GitHub] metron issue #830: METRON-1298: TimeRange Picker doesn't work on Safari

2018-01-30 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/830
  
Can you take master again?  This pr still builds metron twice when doing 
vagrant up.


---


[GitHub] metron issue #684: DO NOT MERGE: METRON-1086: Create a Blockly-based user in...

2018-01-30 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/684
  
We should make this a feature branch


---


[GitHub] metron issue #775: [METRON-1214] rpm build fails due to npm absence in Docke...

2018-01-30 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/775
  
@DimDroll any reproduction?  Is this still happening for you? 


---


[GitHub] metron issue #670: METRON-1065 grok pattern for cisco asa parser expects sys...

2018-01-30 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/670
  
Deconflict and bump?


---


[GitHub] metron issue #890: METRON-1391 Fix for README.md in Metron Management

2018-01-30 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/890
  
Thanks!  please take care of the jira


---


[GitHub] metron issue #858: METRON-1344: Externalize the infrastructural components u...

2018-01-30 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/858
  
Re-upping what I said before


---


[GitHub] metron issue #789: METRON-1233: Remove description of Global configuration f...

2018-01-30 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/789
  
This needs deconflict, and a relevant check 


---


[GitHub] metron issue #912: METRON-1431 Adding REGEX_REPLACE(input, regex, value) to ...

2018-01-29 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/912
  
please remember to take care of your jira


---


[GitHub] metron issue #912: METRON-1431 Adding REGEX_REPLACE(input, regex, value) to ...

2018-01-29 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/912
  
+1 by inspection.  Thank you for the contribution



---


[GitHub] metron pull request #914: METRON-1397 Support for JSON Path and complex docu...

2018-01-29 Thread ottobackwards
GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/914

METRON-1397  Support for JSON Path and complex documents in JSONMapParser

It would be useful for implementors to have the ability to ingest more 
complex documents without having to stand up a NiFi node/cluster.

This PR adds support for splitting multiple messages from a single message 
by way of using JSON Path statements to select the List element in the document.

Since the parse() interface already returns a List the scope of 
this change is in the parser.

Example:

```json
{
"foo" :
[
{ "name" : "foo1", "value" : "bar", "number" : 1.0 },
{ "name" : "foo2", "value" : "baz", "number" : 2.0 }
]
}
 ```

with JSON Path statement "$.foo" will result in two messages emitted from 
the parser

```json
 { "name" : "foo1", "value" : "bar", "number" : 1.0 }
```

```json
{ "name" : "foo2", "value" : "baz", "number" : 2.0 }
```
Support in the configuration for a new configuration parameter: jsonpQuery 
has been added.
An example of the new configuration:

```json
{
  "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
  "sensorTopic":"jsonMapQuery",
  "parserConfig": {"jsonpQuery":"$.foo"}
}
```

In order to integration test both with and without the query present, a new 
jsonMapQuery parser configuration has been added, with test data.


##Testing
- tests should run
- deployment should work


### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:


- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron jsonp-support

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/914.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #914


commit 414611b9ea6eb06b79cba0cc86e63ea062335884
Author: Otto Fowler <ottobackwards@...>
Date:   2018-01-29T16:58:16Z

JSON Path support for JSONMapParser
Tests, Integration Tests, Doc

commit ad03ac4d3e7c98fcf571f3f3906efbb915deb1f9
Author: Otto Fowler <ottobackwards@...>
Date:   2018-01-29T17:13:10Z

checkstyle fixes

commit 7c76c5f5b7507ac55861ed44ad131700c1e6f124
Author: Otto Fowler <ottobackwards@...>
Date:   2018-01-29T19:36:16Z

account for new config in rpm

fix handle empty




---


[GitHub] metron pull request #912: METRON-1431 Adding REGEX_REPLACE(input, regex, val...

2018-01-28 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/912#discussion_r164303375
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/RegExFunctionsTest.java
 ---
@@ -68,4 +68,19 @@ public void testRegExGroupVal() throws Exception {
   Assert.assertTrue("Did not fail on wrong number of 
parameters",false);
 }
   }
+
+  @Test
+  public void testRegExReplace() throws Exception {
+final Map<String, String> variableMap = new HashMap<String, String>() 
{{
+  put("numbers", "12345");
+  put("numberPattern", "\\d(\\d)(\\d).*");
+  put("letters", "abcde");
+  put("empty", "");
+}};
+
--- End diff --

Can you add tests for passing in empty or null strings?


---


[GitHub] metron pull request #912: METRON-1431 Adding REGEX_REPLACE(input, regex, val...

2018-01-28 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/912#discussion_r164303393
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RegExFunctions.java
 ---
@@ -100,4 +100,39 @@ public Object apply(List list) {
   return matcher.group(groupNumber);
 }
   }
+
+  @Stellar(name = "REGEXP_REPLACE",
+  description = "Replace all occurences of the regex pattern within 
the string by value",
+  params = {
+  "string - The input string",
+  "pattern - The regex pattern to be replaced. Special characters 
must be escaped (e.g. d)",
+  "value - The value to replace the regex pattern"
+  },
+  returns = "The modified input string with replaced values")
+  public static class RegexpReplace extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+  if (list.size() != 3) {
+throw new IllegalStateException(
+"REGEXP_REPLACE expects three args: [string, pattern, value]"
++ " where pattern is a regexp pattern");
+  }
+  String str = (String) list.get(0);
+  String stringPattern = (String) list.get(1);
+  String value = (String) list.get(2);
+
--- End diff --

I think you want to check for null or empty

StringUtils.isEmpty(str)


---


[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

2018-01-26 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/903
  
+1 still stands



---


[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

2018-01-25 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/903
  
+1 -> ran up both images, everything checked out
Ship it


---


[GitHub] metron issue #901: METRON-1410 [MPACK] Check for existing HBASE tables befor...

2018-01-25 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/901
  
I'm going to wait for @lvets a chance to try his scenario


---


[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

2018-01-25 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/903
  
Sure.  
Question.  Do we expect there to be issues with 2.6?  Is this PR and 
Casey's 2.6 pr going to conflict or have issues?  How will we know to retest 
this after that one lands if this lands first? etc etc


---


[GitHub] metron issue #902: METRON-1413 Add Metron Commit Tool

2018-01-25 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/902
  
+1


---


[GitHub] metron issue #910: METRON-1430: Isolate jackson from being used as arguments...

2018-01-25 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/910
  
This looks awesome at first look.  I'm a big fan of doing this regardless 
of the shading issue.  The only thought that comes to mind is that there is a 
tipping point where a *Utils class isn't the right thing, that is it outgrown.  
I feel like we might be there for JSONUtils.

We can do that in another pr however I guess.


---


[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

2018-01-25 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/903
  
@nickwallen let me know when you feel ok about it, I'll run it through 
again.


---


[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

2018-01-25 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/903
  
Failure during vagrant up for metron-on-ubuntu

```

2018-01-25 10:41:49,302 p=37541 u=ottofowler |  fatal: [node1]: 
FAILED! => {"changed": true, "cmd": "dpkg-scanpackages . /dev/null | gzip -9c > 
dists/METRON/main/binary-amd64/Packages.gz", "delta": "0:00:00.062622", "end": 
"2018-01-25 15:41:49.325958", "failed": true, "rc": 2, "start": "2018-01-25 
15:41:49.263336", "stderr": "/bin/sh: 1: cannot create 
dists/METRON/main/binary-amd64/Packages.gz: Directory 
nonexistent\ndpkg-scanpackages: info: Wrote 0 entries to output Packages 
file.", "stdout": "", "stdout_lines": [], "warnings": []}
2018-01-25 10:41:49,332 p=37541 u=ottofowler |    
< PLAY RECAP >
  
\   ^__^
 \  (oo)\___
(__)\   )\/\
||w |
|| ||
```


--
platform_info
--

```
Metron 0.4.3
--
* pr-903
--
commit d2cf8fca52d746ec65597cbb826f4cb9ac886a21
Author: Nick Allen <n...@nickallen.org>
Date:   Wed Jan 24 19:07:49 2018 -0500

The sensors and pcap replay need to start themselves since Monit will 
not do it
--
--
ansible 2.0.0.2
  config file = 
/Users/ottofowler/tmp/metron-pr-903/metron-deployment/vagrant/metron-on-ubuntu/ansible.cfg
  configured module search path = ../../ansible/extra_modules
--
Vagrant 2.0.1
--
Python 2.7.14
--
Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 
2017-10-18T03:58:13-04:00)
Maven home: /usr/local/Cellar/maven/3.5.2/libexec
Java version: 1.8.0_121, vendor: Oracle Corporation
Java home: 
/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.13.3", arch: "x86_64", family: "mac"
--
Docker version 17.12.0-ce, build c97c6d6
--
node
v6.10.2
--
npm
3.10.10
--
Apple LLVM version 9.0.0 (clang-900.0.39.2)
Target: x86_64-apple-darwin17.4.0
Thread model: posix
InstalledDir: 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
--
Compiler is C++11 compliant
--
Darwin Winterfell 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 
PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64
--
Total System Memory = 16384 MB
Processor Model: Intel(R) Core(TM) i7-4870HQ CPU 
Processor Speed: 2.50GHz
Total Physical Processors: 4
Total cores: 4
Disk information:
/dev/disk1s1   465Gi  380Gi   81Gi83% 3929490 92233720368508463170% 
  /
/dev/disk1s4   465Gi  3.0Gi   81Gi 4%   3 92233720368547758040% 
  /private/var/vm
This CPU appears to support virtualization
```


---


[GitHub] metron issue #903: METRON-1370 Create Full Dev Equivalent for Ubuntu

2018-01-25 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/903
  
Trying this now.   
Only comment on the content here is there is a _lot_ going on in this pr.  
A lot of while I'm here I might as well work.
It might have been better to have kept this more narrow ( although it was 
never going to be narrow ).
I wouldn't change anything at this point however.


---


[GitHub] metron pull request #901: METRON-1410 Check for existing HBASE tables before...

2018-01-25 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/901#discussion_r163868868
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
 ---
@@ -190,7 +190,7 @@ def create_hbase_tables(self):
   self.__params.hbase_principal_name,
   execute_user=self.__params.hbase_user)
 
-cmd = "echo \"create '{0}','{1}'\" | hbase shell -n"
+cmd = "if [[ $(echo \"exists '{0}'\" | hbase shell | grep 'not 
exist') ]]; then echo \"create '{0}','{1}'\" | hbase shell -n; fi"
--- End diff --

done



---


[GitHub] metron issue #902: METRON-1413 Add Metron Commit Tool

2018-01-25 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/902
  
I have another script I want to add for tracking master in feature branches 
after this as well


---


[GitHub] metron issue #902: METRON-1413 Add Metron Commit Tool

2018-01-24 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/902
  
I can almost always be bought off with a jira number


---


[GitHub] metron issue #888: METRON-1389: Zeppelin notebook import does not work with ...

2018-01-24 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/888
  
+1 by inspection


---


[GitHub] metron pull request #901: METRON-1410 Check for existing HBASE tables before...

2018-01-24 Thread ottobackwards
Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/901#discussion_r163616902
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
 ---
@@ -190,7 +190,7 @@ def create_hbase_tables(self):
   self.__params.hbase_principal_name,
   execute_user=self.__params.hbase_user)
 
-cmd = "echo \"create '{0}','{1}'\" | hbase shell -n"
+cmd = "if [[ $(echo \"exists '{0}'\" | hbase shell | grep 'not 
exist') ]]; then echo \"create '{0}','{1}'\" | hbase shell -n; fi"
--- End diff --

ok, i'll do that


---


[GitHub] metron issue #873: METRON-1367 Stellar should have some instrumentation of f...

2018-01-22 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/873
  
ok, it is MAGIC_TIME! @simonellistonball 


---


[GitHub] metron issue #873: METRON-1367 Stellar should have some instrumentation of f...

2018-01-22 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/873
  
Plus it lets me get away with 

```java
public static final String MAGIC_TIME = "%time";
```

so bonus


---


[GitHub] metron issue #873: METRON-1367 Stellar should have some instrumentation of f...

2018-01-22 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/873
  
well, %timing seem to make sense to me, as it is the timing of the last 
statement. %time didn't occur to me.  I don't use Jupyter, and don't time 
things in the shell that often.  

I am didn't really canvas other tools.
I will change it.


---


[GitHub] metron issue #690: METRON-1091 Package STELLAR shell as stand alone

2018-01-22 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/690
  
@JonZeolla  this should be set


---


[GitHub] metron issue #870: METRON-1364: Add an implementation of Robust PCA outlier ...

2018-01-22 Thread ottobackwards
Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/870
  
Looking for tests and deconflict


---


  1   2   3   4   5   6   7   8   9   10   >