[jira] [Commented] (METRON-455) Create stellar management functions to put, get and list data in HDFS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524705#comment-15524705
 ] 

ASF GitHub Bot commented on METRON-455:
---

GitHub user cestella opened a pull request:

https://github.com/apache/incubator-metron/pull/277

METRON-455: Create stellar management functions to put, get and list data 
in HDFS

Create functions around reading, writing and removing local and HDFS files.

Functions added:
* Interacting with HDFS
  * `HDFS_LS`
  * `HDFS_RM`
  * `HDFS_GET`
  * `HDFS_PUT`
* Interacting with the local filesystem
  * `FILE_LS`
  * `FILE_RM`
  * `FILE_GET`
  * `FILE_PUT`

You can test these by deploying the management jar as per the instructions 
in `metron-management` and try them out in the Stellar REPL.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron METRON-455

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/277.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #277


commit e4cd0620c64bc67bde027caa224b99dbaf454636
Author: cstella 
Date:   2016-09-26T22:01:01Z

Initial implementation

commit 58fda38fa5616e876254f5f535c86518bc805b48
Author: cstella 
Date:   2016-09-26T22:01:27Z

Merge branch 'master' into METRON-455

commit 57897decf4a8eec03d6e32ec7fcaab54db8d8563
Author: cstella 
Date:   2016-09-27T01:15:52Z

Added unit/integration tests.




> Create stellar management functions to put, get and list data in HDFS
> -
>
> Key: METRON-455
> URL: https://issues.apache.org/jira/browse/METRON-455
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Casey Stella
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-453) Add a stellar shell function to open an external editor and return the editor's contents

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524314#comment-15524314
 ] 

ASF GitHub Bot commented on METRON-453:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/272


> Add a stellar shell function to open an external editor and return the 
> editor's contents
> 
>
> Key: METRON-453
> URL: https://issues.apache.org/jira/browse/METRON-453
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
> Fix For: 0.2.1BETA
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> In order to create multiline strings (e.g. JSON or grok statements), we 
> should have a stellar shell function which can open up an editor (specified 
> by the `EDITOR` or `VISUAL` environment variable) optionally initialized by 
> the contents of a variable and return the output of the editor when saved and 
> closed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-453) Add a stellar shell function to open an external editor and return the editor's contents

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524137#comment-15524137
 ] 

ASF GitHub Bot commented on METRON-453:
---

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/272
  
+1 tested on quickdev and was able to use SHELL_EDIT with and without the 
optional param and with various input and on both vi and nano as editors.


> Add a stellar shell function to open an external editor and return the 
> editor's contents
> 
>
> Key: METRON-453
> URL: https://issues.apache.org/jira/browse/METRON-453
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
> Fix For: 0.2.1BETA
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> In order to create multiline strings (e.g. JSON or grok statements), we 
> should have a stellar shell function which can open up an editor (specified 
> by the `EDITOR` or `VISUAL` environment variable) optionally initialized by 
> the contents of a variable and return the output of the editor when saved and 
> closed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-411) Support Greater Range of Profile Periods

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523976#comment-15523976
 ] 

ASF GitHub Bot commented on METRON-411:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/275
  
I'm +1 on the change; good catch.  Could you please make a JIRA for this 
and rename this PR to ${JIRA}: ${JIRA DESCRIPTION}.  For instance, if the JIRA 
is METRON-411 adn the description is "Fixed package error", could you call the 
title "METRON-411: Fixed package error"?


> Support Greater Range of Profile Periods 
> -
>
> Key: METRON-411
> URL: https://issues.apache.org/jira/browse/METRON-411
> Project: Metron
>  Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> The current Profiler's deterministic row key design limits the range of valid 
> profile period durations.  The original goal was to support 15 minute periods 
> which it does, along with 5, 20, 30 and 60 minute periods.  Different types 
> of profiles and input data will require a range of periods and the Profiler 
> should be flexible in this regards.
> Currently, a profile period must start on each and every hour.  While 1, 2, 
> 3, 4, 6, or 240 periods per hour is acceptable, 5 periods per hour is not.  
> The number of periods per hour must be either a multiple or divisor of 60.  
> Also, a period cannot be any longer than an hour.  In some cases 2, 4, or 8 
> hour periods might be useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-457) Typo in grok parser debug message needs to be corrected

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523960#comment-15523960
 ] 

ASF GitHub Bot commented on METRON-457:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/274
  
+1


> Typo in grok parser debug message needs to be corrected
> ---
>
> Key: METRON-457
> URL: https://issues.apache.org/jira/browse/METRON-457
> Project: Metron
>  Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Minor
>
> GrokParser log statements misspelled "parser" as "perser".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-452) Add rudimentary configuration management functions to Stellar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523926#comment-15523926
 ] 

ASF GitHub Bot commented on METRON-452:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/269


> Add rudimentary configuration management functions to Stellar
> -
>
> Key: METRON-452
> URL: https://issues.apache.org/jira/browse/METRON-452
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
> Fix For: 0.2.1BETA
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Currently, the only way to add enrichments is via editing JSON files and 
> pushing them to zookeeper.  Rather than that, because we have a REPL, we 
> should create management functions around:
> * Shell functions - Functions surrounding interacting with the shell in 
> either a nicer way or a more functional way.
> * Configuration functions - Functions surrounding pulling and pushing configs 
> from zookeeper
> * Parser functions - Functions surrounding adding, viewing, and removing 
> Parser functions.
> * Enrichment functions - Functions surrounding adding, viewing and removing 
> Stellar enrichments as well as managing batch size and index names for the 
> enrichment topology   configuration
> * Threat Triage functions - Functions surrounding adding, viewing and 
> removing threat triage functions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-452) Add rudimentary configuration management functions to Stellar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523772#comment-15523772
 ] 

ASF GitHub Bot commented on METRON-452:
---

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/269
  
I'm +1, pending the Travis check


> Add rudimentary configuration management functions to Stellar
> -
>
> Key: METRON-452
> URL: https://issues.apache.org/jira/browse/METRON-452
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
> Fix For: 0.2.1BETA
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Currently, the only way to add enrichments is via editing JSON files and 
> pushing them to zookeeper.  Rather than that, because we have a REPL, we 
> should create management functions around:
> * Shell functions - Functions surrounding interacting with the shell in 
> either a nicer way or a more functional way.
> * Configuration functions - Functions surrounding pulling and pushing configs 
> from zookeeper
> * Parser functions - Functions surrounding adding, viewing, and removing 
> Parser functions.
> * Enrichment functions - Functions surrounding adding, viewing and removing 
> Stellar enrichments as well as managing batch size and index names for the 
> enrichment topology   configuration
> * Threat Triage functions - Functions surrounding adding, viewing and 
> removing threat triage functions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-452) Add rudimentary configuration management functions to Stellar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523475#comment-15523475
 ] 

ASF GitHub Bot commented on METRON-452:
---

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/269#discussion_r80511265
  
--- Diff: 
metron-platform/metron-management/src/main/java/org/apache/metron/management/ParserConfigFunctions.java
 ---
@@ -0,0 +1,215 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.management;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import com.jakewharton.fliptables.FlipTable;
+import org.apache.log4j.Logger;
+import org.apache.metron.common.configuration.ConfigurationType;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.common.dsl.*;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.FieldTransformations;
+import org.apache.metron.common.stellar.shell.StellarExecutor;
+import org.apache.metron.common.utils.JSONUtils;
+import org.jboss.aesh.console.Console;
+
+import java.util.*;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+public class ParserConfigFunctions {
+  private static final Logger LOG = 
Logger.getLogger(ConfigurationFunctions.class);
+
+  private static void pruneEmptyStellarTransformers(SensorParserConfig 
config) {
+List toRemove = new ArrayList<>();
+List fieldTransformations = 
config.getFieldTransformations();
+for(FieldTransformer transformer : fieldTransformations) {
+  if(transformer.getFieldTransformation().getClass().getName()
+  
.equals(FieldTransformations.STELLAR.getMappingClass().getName())) {
+if(transformer == null || transformer.getConfig().isEmpty()) {
+  toRemove.add(transformer);
+}
+  }
+}
+for(FieldTransformer t : toRemove) {
+  fieldTransformations.remove(t);
+}
+  }
+  private static FieldTransformer getStellarTransformer(SensorParserConfig 
config) {
+List fieldTransformations = 
config.getFieldTransformations();
+FieldTransformer stellarTransformer = null;
+for(FieldTransformer transformer : fieldTransformations) {
+  if(transformer.getFieldTransformation().getClass().getName()
+  
.equals(FieldTransformations.STELLAR.getMappingClass().getName())) {
+stellarTransformer = transformer;
+  }
+}
+if(stellarTransformer == null) {
+  stellarTransformer = new FieldTransformer();
+  stellarTransformer.setConfig(new LinkedHashMap<>());
+  
stellarTransformer.setTransformation(FieldTransformations.STELLAR.toString());
+  fieldTransformations.add(stellarTransformer);
+}
+return stellarTransformer;
+  }
+
+  @Stellar(
+   namespace = "PARSER_STELLAR_TRANSFORM"
+  ,name = "PRINT"
+  ,description = "Retrieve stellar field transformations."
+  ,params = {"sensorConfig - Sensor config to add transformation 
to."
+}
+  ,returns = "The String representation of the transformations"
+  )
+  public static class PrintStellarTransformation implements 
StellarFunction {
+
+@Override
+public Object apply(List args, Context context) throws 
ParseException {
+  String config = (String) args.get(0);
+  if(config == null) {
+return null;
+  }
+  SensorParserConfig configObj = (SensorParserConfig) 
PARSER.deserialize(config);
+  FieldTransformer stellarTransformer = 

[jira] [Commented] (METRON-449) JSON Parser should handle arbitrarily deep nesting of maps

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523459#comment-15523459
 ] 

ASF GitHub Bot commented on METRON-449:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/271
  
Never mind - just found the field name converter


> JSON Parser should handle arbitrarily deep nesting of maps
> --
>
> Key: METRON-449
> URL: https://issues.apache.org/jira/browse/METRON-449
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.2.1BETA
>Reporter: Otto Fowler
>
> The JSONMapParser's unfolding function only unfolds maps at the root level
> such as "collection" : { "blah" : 7, "blah2" : "foo" }.
> This limits the utility of the parser, since it is likely that someone would 
> have to do a good deal of transformation before hand.
> The ability for the parser to handle nested objects ( maps ) to arbitrary 
> depth would greatly add to the utility of this parser.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-449) JSON Parser should handle arbitrarily deep nesting of maps

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523458#comment-15523458
 ] 

ASF GitHub Bot commented on METRON-449:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/271
  
No, ES uses ':' and modifies the keys upon write converting '.' to ':'.
Before that, '.' is the preferred source.  I'm hoping we can go back to '.'
eventually.  Also, non-ES indices are perfectly fine with '.'.

On Mon, Sep 26, 2016 at 11:59 AM, ottobackwards 
wrote:

> @cestella  , from looking at the ES
> templates etc, it seems that ':' is used as the name separator and not 
'.'.
> Should the the unfolding not use the same?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> 
,
> or mute the thread
> 

> .
>



> JSON Parser should handle arbitrarily deep nesting of maps
> --
>
> Key: METRON-449
> URL: https://issues.apache.org/jira/browse/METRON-449
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.2.1BETA
>Reporter: Otto Fowler
>
> The JSONMapParser's unfolding function only unfolds maps at the root level
> such as "collection" : { "blah" : 7, "blah2" : "foo" }.
> This limits the utility of the parser, since it is likely that someone would 
> have to do a good deal of transformation before hand.
> The ability for the parser to handle nested objects ( maps ) to arbitrary 
> depth would greatly add to the utility of this parser.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-449) JSON Parser should handle arbitrarily deep nesting of maps

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523452#comment-15523452
 ] 

ASF GitHub Bot commented on METRON-449:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/271
  
@cestella , from looking at the ES templates etc, it seems that ':' is used 
as the name separator and not '.'.  Should the the unfolding not use the same?


> JSON Parser should handle arbitrarily deep nesting of maps
> --
>
> Key: METRON-449
> URL: https://issues.apache.org/jira/browse/METRON-449
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.2.1BETA
>Reporter: Otto Fowler
>
> The JSONMapParser's unfolding function only unfolds maps at the root level
> such as "collection" : { "blah" : 7, "blah2" : "foo" }.
> This limits the utility of the parser, since it is likely that someone would 
> have to do a good deal of transformation before hand.
> The ability for the parser to handle nested objects ( maps ) to arbitrary 
> depth would greatly add to the utility of this parser.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (METRON-410) mysql_server's MySQL install causes mutually assured destruction when installed on the same machine as the Ambari Hive MySQL

[Justin Leet (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Justin Leet updated METRON-410:
---
Issue Type: Improvement  (was: Sub-task)
Parent: (was: METRON-427)

> mysql_server's MySQL install causes mutually assured destruction when 
> installed on the same machine as the Ambari Hive MySQL
> 
>
> Key: METRON-410
> URL: https://issues.apache.org/jira/browse/METRON-410
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>Priority: Minor
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> Metron's mysql_server MySQL install causes mutually assured destruction when 
> installed on the same machine as the Ambari Hive MySQL.  Here is the startup 
> error you get afterwards.  
> https://gist.github.com/JonZeolla/2ed4161c141ba32a3e8a0d6ce9718779
> In the short term, maybe add a check so they won't live on the same box.  
> Long term, allow them to coexist?  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-457) Typo in grok parser debug message needs to be corrected

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523390#comment-15523390
 ] 

ASF GitHub Bot commented on METRON-457:
---

GitHub user mmiklavc opened a pull request:

https://github.com/apache/incubator-metron/pull/274

METRON-457 Correct GrokParser logging spelling error

In GrokParser.java
`%s/persers/parsers/g`


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mmiklavc/incubator-metron METRON-457

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/274.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #274


commit a05f33c737c8c2b43979fbb9ace3181a6bed3fb5
Author: Michael Miklavcic 
Date:   2016-09-26T15:09:02Z

METRON-457 Correct logging spelling error




> Typo in grok parser debug message needs to be corrected
> ---
>
> Key: METRON-457
> URL: https://issues.apache.org/jira/browse/METRON-457
> Project: Metron
>  Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Minor
>
> GrokParser log statements misspelled "parser" as "perser".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-452) Add rudimentary configuration management functions to Stellar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523381#comment-15523381
 ] 

ASF GitHub Bot commented on METRON-452:
---

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/269#discussion_r80502328
  
--- Diff: 
metron-platform/metron-management/src/main/java/org/apache/metron/management/EnrichmentConfigFunctions.java
 ---
@@ -0,0 +1,351 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.management;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.jakewharton.fliptables.FlipTable;
+import org.apache.log4j.Logger;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.utils.ConversionUtils;
+import org.apache.metron.common.utils.JSONUtils;
+
+import java.util.*;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.ENRICHMENT;
+
+public class EnrichmentConfigFunctions {
+
+  private static final Logger LOG = 
Logger.getLogger(ConfigurationFunctions.class);
+  public enum Type {
+ENRICHMENT, THREAT_INTEL, THREATINTEL;
+  }
+  public static Map getStellarHandler(EnrichmentConfig 
enrichmentConfig) {
+Map fieldMap = enrichmentConfig.getFieldMap();
+Map stellarHandler = (Map) 
fieldMap.get("stellar");
+if(stellarHandler == null ) {
+  stellarHandler = new HashMap();
+  fieldMap.put("stellar", stellarHandler);
+}
+
+if(stellarHandler.get("config") == null){
--- End diff --

Can we use `stellarHandler.putIfAbsent("config", new LinkedHashMap());`?  It's a Java 8 thing, and I haven't seen anyone suggest moving 
back to Java 7, so might as well get some use out of those functions.


> Add rudimentary configuration management functions to Stellar
> -
>
> Key: METRON-452
> URL: https://issues.apache.org/jira/browse/METRON-452
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
> Fix For: 0.2.1BETA
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Currently, the only way to add enrichments is via editing JSON files and 
> pushing them to zookeeper.  Rather than that, because we have a REPL, we 
> should create management functions around:
> * Shell functions - Functions surrounding interacting with the shell in 
> either a nicer way or a more functional way.
> * Configuration functions - Functions surrounding pulling and pushing configs 
> from zookeeper
> * Parser functions - Functions surrounding adding, viewing, and removing 
> Parser functions.
> * Enrichment functions - Functions surrounding adding, viewing and removing 
> Stellar enrichments as well as managing batch size and index names for the 
> enrichment topology   configuration
> * Threat Triage functions - Functions surrounding adding, viewing and 
> removing threat triage functions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-452) Add rudimentary configuration management functions to Stellar

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523367#comment-15523367
 ] 

ASF GitHub Bot commented on METRON-452:
---

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/269#discussion_r80500701
  
--- Diff: 
metron-platform/metron-management/src/main/java/org/apache/metron/management/ParserConfigFunctions.java
 ---
@@ -0,0 +1,215 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.management;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import com.jakewharton.fliptables.FlipTable;
+import org.apache.log4j.Logger;
+import org.apache.metron.common.configuration.ConfigurationType;
+import org.apache.metron.common.configuration.FieldTransformer;
+import org.apache.metron.common.configuration.SensorParserConfig;
+import org.apache.metron.common.dsl.*;
+import org.apache.metron.common.field.transformation.FieldTransformation;
+import org.apache.metron.common.field.transformation.FieldTransformations;
+import org.apache.metron.common.stellar.shell.StellarExecutor;
+import org.apache.metron.common.utils.JSONUtils;
+import org.jboss.aesh.console.Console;
+
+import java.util.*;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.PARSER;
+
+public class ParserConfigFunctions {
+  private static final Logger LOG = 
Logger.getLogger(ConfigurationFunctions.class);
+
+  private static void pruneEmptyStellarTransformers(SensorParserConfig 
config) {
+List toRemove = new ArrayList<>();
+List fieldTransformations = 
config.getFieldTransformations();
+for(FieldTransformer transformer : fieldTransformations) {
+  if(transformer.getFieldTransformation().getClass().getName()
--- End diff --

This set of if statements seems questionable.  First, should the 
`transformer == null` be moved up a level?  It already gets dereferenced, so 
the check is either unnecessary or should be moved up a level.  At that point, 
I question why there are nested ifs in the first place (if it's anything other 
than for clarity, given the length of the first if).


> Add rudimentary configuration management functions to Stellar
> -
>
> Key: METRON-452
> URL: https://issues.apache.org/jira/browse/METRON-452
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
> Fix For: 0.2.1BETA
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Currently, the only way to add enrichments is via editing JSON files and 
> pushing them to zookeeper.  Rather than that, because we have a REPL, we 
> should create management functions around:
> * Shell functions - Functions surrounding interacting with the shell in 
> either a nicer way or a more functional way.
> * Configuration functions - Functions surrounding pulling and pushing configs 
> from zookeeper
> * Parser functions - Functions surrounding adding, viewing, and removing 
> Parser functions.
> * Enrichment functions - Functions surrounding adding, viewing and removing 
> Stellar enrichments as well as managing batch size and index names for the 
> enrichment topology   configuration
> * Threat Triage functions - Functions surrounding adding, viewing and 
> removing threat triage functions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-374) Add appropriate bundled 3rd party licenses to NOTICE and LICENSE where appropriate

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523365#comment-15523365
 ] 

ASF GitHub Bot commented on METRON-374:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/229


> Add appropriate bundled 3rd party licenses to NOTICE and LICENSE where 
> appropriate
> --
>
> Key: METRON-374
> URL: https://issues.apache.org/jira/browse/METRON-374
> Project: Metron
>  Issue Type: Task
>Reporter: Casey Stella
> Fix For: 0.2.1BETA
>
>
> We are careful around licensing source bundled components appropriately, but 
> we have neglected to handle binary bundled components as part of the uber 
> jars that we create.  These 3rd party dependencies should be referenced in 
> the NOTICE and LICENSE files where appropriate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-427) Create Ambari Management Pack for Metron Installation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523346#comment-15523346
 ] 

ASF GitHub Bot commented on METRON-427:
---

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/266


> Create Ambari Management Pack for Metron Installation
> -
>
> Key: METRON-427
> URL: https://issues.apache.org/jira/browse/METRON-427
> Project: Metron
>  Issue Type: New Feature
>Reporter: Justin Leet
>Assignee: Justin Leet
>
> Right now, Metron depends on Ambari blueprints, in the Ansible scripts, to 
> deploy onto a cluster.
> To ease installation, a full Ambari Management Pack 
> (https://cwiki.apache.org/confluence/display/AMBARI/Management+Packs) can be 
> used to lay down topologies, etc.
> The current expectation is that the boundaries of this would cover from Kafka 
> to the indexes.  The dev list has additional discussion about whether or not 
> sensor install and what should exist beyond minimum viable product.  
> Additional follow-on tickets would be created based on the results of both 
> that discussion and any discussion on this ticket.
> A minimum viable product for this would cover
> * Laying down topologies (parsers, enrichment, and indexing)
> * Starting and stopping topologies
> * Setting up configuration
> * Setting up bits (Using RPMs currently built locally)
> * Set up infra dependencies (MySql and Elasticsearch)
> At this point, the MVP could take data from Kafka, run it through the 
> topologies, and make it available in the output Elasticsearch Indexes.
> A good deal of the ground work for this is already completed (several Service 
> Definitions, along with the RPM creation).  Relevant Jira's are:
> * METRON-383 (Create Ambari Service Definition for Metron Parsers)
> * METRON-385 (Create Ambari Service Definition for Indexing)
> * METRON-386 (Create Ambari Service Definition for Elasticsearch)
> * METRON-357 (Create Ambari Service Definition for Kibana)
> * METRON-214 (Build RPM Packages for Deployment)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (METRON-457) Typo in grok parser debug message needs to be corrected

[Michael Miklavcic (JIRA)]

Michael Miklavcic created METRON-457:


 Summary: Typo in grok parser debug message needs to be corrected
 Key: METRON-457
 URL: https://issues.apache.org/jira/browse/METRON-457
 Project: Metron
  Issue Type: Bug
Reporter: Michael Miklavcic
Assignee: Michael Miklavcic
Priority: Minor


GrokParser log statements misspelled "parser" as "perser".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (METRON-410) mysql_server's MySQL install causes mutually assured destruction when installed on the same machine as the Ambari Hive MySQL

[David M. Lyle (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David M. Lyle updated METRON-410:
-
Issue Type: Sub-task  (was: Bug)
Parent: METRON-427

> mysql_server's MySQL install causes mutually assured destruction when 
> installed on the same machine as the Ambari Hive MySQL
> 
>
> Key: METRON-410
> URL: https://issues.apache.org/jira/browse/METRON-410
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Priority: Minor
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> Metron's mysql_server MySQL install causes mutually assured destruction when 
> installed on the same machine as the Ambari Hive MySQL.  Here is the startup 
> error you get afterwards.  
> https://gist.github.com/JonZeolla/2ed4161c141ba32a3e8a0d6ce9718779
> In the short term, maybe add a check so they won't live on the same box.  
> Long term, allow them to coexist?  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (METRON-405) Create Ambari Service Definition for Enrichment

[David M. Lyle (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David M. Lyle updated METRON-405:
-
Issue Type: Sub-task  (was: New Feature)
Parent: METRON-427

> Create Ambari Service Definition for Enrichment
> ---
>
> Key: METRON-405
> URL: https://issues.apache.org/jira/browse/METRON-405
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>
> To pull everything into an easier install through Ambari, create a service 
> definition to automatically install and handle the enrichment topology 
> appropriately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-427) Create Ambari Management Pack for Metron Installation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522978#comment-15522978
 ] 

ASF GitHub Bot commented on METRON-427:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/266
  
+1 by inspection



> Create Ambari Management Pack for Metron Installation
> -
>
> Key: METRON-427
> URL: https://issues.apache.org/jira/browse/METRON-427
> Project: Metron
>  Issue Type: New Feature
>Reporter: Justin Leet
>Assignee: Justin Leet
>
> Right now, Metron depends on Ambari blueprints, in the Ansible scripts, to 
> deploy onto a cluster.
> To ease installation, a full Ambari Management Pack 
> (https://cwiki.apache.org/confluence/display/AMBARI/Management+Packs) can be 
> used to lay down topologies, etc.
> The current expectation is that the boundaries of this would cover from Kafka 
> to the indexes.  The dev list has additional discussion about whether or not 
> sensor install and what should exist beyond minimum viable product.  
> Additional follow-on tickets would be created based on the results of both 
> that discussion and any discussion on this ticket.
> A minimum viable product for this would cover
> * Laying down topologies (parsers, enrichment, and indexing)
> * Starting and stopping topologies
> * Setting up configuration
> * Setting up bits (Using RPMs currently built locally)
> * Set up infra dependencies (MySql and Elasticsearch)
> At this point, the MVP could take data from Kafka, run it through the 
> topologies, and make it available in the output Elasticsearch Indexes.
> A good deal of the ground work for this is already completed (several Service 
> Definitions, along with the RPM creation).  Relevant Jira's are:
> * METRON-383 (Create Ambari Service Definition for Metron Parsers)
> * METRON-385 (Create Ambari Service Definition for Indexing)
> * METRON-386 (Create Ambari Service Definition for Elasticsearch)
> * METRON-357 (Create Ambari Service Definition for Kibana)
> * METRON-214 (Build RPM Packages for Deployment)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)