[jira] [Commented] (METRON-1442) Rest endpoints for operations on Indexing topology is broken after splitting the indexing topology

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351218#comment-16351218
 ] 

ASF GitHub Bot commented on METRON-1442:


Github user MohanDV commented on the issue:

https://github.com/apache/metron/pull/923
  
@cestella I merged your PR


> Rest endpoints for operations on Indexing topology is broken after splitting 
> the indexing topology
> --
>
> Key: METRON-1442
> URL: https://issues.apache.org/jira/browse/METRON-1442
> Project: Metron
>  Issue Type: Bug
>Reporter: Mohan
>Assignee: Mohan
>Priority: Major
>
> Rest endpoints for operations on Indexing topology is broken after splitting 
> the indexing topology into random_access_indexing and batch_indexing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1442) Rest endpoints for operations on Indexing topology is broken after splitting the indexing topology

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351041#comment-16351041
 ] 

ASF GitHub Bot commented on METRON-1442:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/923
  
Looks like you didn't quite get all of the mock test infrastructure for 
`metron-rest` set up properly.  I went ahead and submitted a PR against your 
branch to help out. :) If you can merge 
https://github.com/MohanDV/metron/pull/1 into your branch, it should fix the 
tests.


> Rest endpoints for operations on Indexing topology is broken after splitting 
> the indexing topology
> --
>
> Key: METRON-1442
> URL: https://issues.apache.org/jira/browse/METRON-1442
> Project: Metron
>  Issue Type: Bug
>Reporter: Mohan
>Assignee: Mohan
>Priority: Major
>
> Rest endpoints for operations on Indexing topology is broken after splitting 
> the indexing topology into random_access_indexing and batch_indexing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1441) Create complementary Solr schemas for the main sensors

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351037#comment-16351037
 ] 

ASF GitHub Bot commented on METRON-1441:


Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/922
  
I tested this in full dev using the install script in 
https://github.com/apache/metron/pull/918.  I was able to create collections 
for each schema except for "error".  For that to work properly, I had to: 

- remove `docValues="true"` from the "bytes" field type
- add the "guid" field used in other schemas

Still working on indexing data into these collections but so far so good.


> Create complementary Solr schemas for the main sensors
> --
>
> Key: METRON-1441
> URL: https://issues.apache.org/jira/browse/METRON-1441
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> We have ES templates for bro, snort, yaf, and error, we need corresponding 
> solr schemas for these collections.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1443) Missing Critical MPack Install Instruction for Ubuntu

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350999#comment-16350999
 ] 

ASF GitHub Bot commented on METRON-1443:


GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/925

METRON-1443 Missing Critical MPack Install Instruction for Ubuntu

When installing Elasticsearch with the MPack on Ubuntu, you must manually 
install the Elasticsearch repositories.  The Mpack itself does not do this, 
like it does on CentOS. 

When the development environment on Ubuntu is spun-up this step is 
performed within Ansible as a prerequisite to the Mpack install.  Until this 
can be fixed so that it matches what happens on CentOS, this needs to be at 
least documented.

I should have documented this in #903 , but did not do so.  Oops.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1443

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/925.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #925


commit 0c40178494d2a12e8e4abbd43ba4f85338aa05da
Author: Nick Allen 
Date:   2018-02-02T22:12:23Z

METRON-1443 Missing Critical MPack Install Instruction for Ubuntu




> Missing Critical MPack Install Instruction for Ubuntu
> -
>
> Key: METRON-1443
> URL: https://issues.apache.org/jira/browse/METRON-1443
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> When installing Elasticsearch with the MPack on Ubuntu, you must manually 
> install the Elasticsearch repositories.  The Mpack itself does not do this, 
> like it does on CentOS. 
> When the development environment on Ubuntu is spun-up this step is performed 
> within Ansible as a prerequisite to the Mpack install.  Until this can be 
> fixed so that it matches what happens in CentOS this needs to be at least 
> documented.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1443) Missing Critical MPack Install Instruction for Ubuntu

2018-02-02 Thread Nick Allen (JIRA) 
Nick Allen created METRON-1443:
--

 Summary: Missing Critical MPack Install Instruction for Ubuntu
 Key: METRON-1443
 URL: https://issues.apache.org/jira/browse/METRON-1443
 Project: Metron
  Issue Type: Bug
Reporter: Nick Allen
Assignee: Nick Allen


When installing Elasticsearch with the MPack on Ubuntu, you must manually 
install the Elasticsearch repositories.  The Mpack itself does not do this, 
like it does on CentOS. 

When the development environment on Ubuntu is spun-up this step is performed 
within Ansible as a prerequisite to the Mpack install.  Until this can be fixed 
so that it matches what happens in CentOS this needs to be at least documented.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1299) MetronError tests fail if hostname isn't set

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350995#comment-16350995
 ] 

ASF GitHub Bot commented on METRON-1299:


GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/924

METRON-1299 In MetronError tests, don't test for HostName if getHostName 
wouldn't work

MetronError ignores exceptions from 
InetAddress.getLocalHost().getHostName() and leaves the field unset.

The unit test however assumes it would be set, and someone has logged a 
jira on this, since it makes the build fail.

Changed the test so that it only verifies the hostName if it would have 
worked.

### Testing
- Code review
- Tests Pass

> no non-test changes in pr

```java
 private void addHostname(JSONObject errorMessage) {
try {
  errorMessage.put(ErrorFields.HOSTNAME.getName(), 
InetAddress.getLocalHost().getHostName());
} catch (UnknownHostException ex) {
  // Leave the hostname field off if it cannot be found
}
  }
```

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron error_addHost

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/924.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #924


commit cee3acba914f97ca7d2faf6e7822c97928a4e242
Author: Otto Fowler 
Date:   2018-02-02T21:57:47Z

do not test for hostName if calling hostName throws, since it will be null




> MetronError tests fail if hostname isn't set
> 
>
> Key: METRON-1299
> URL: https://issues.apache.org/jira/browse/METRON-1299
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
> Environment: openSUSE Tumbleweed 20171102, OpenJDK 1.8.0_144
>Reporter: Stuart Bertram
>Assignee: Otto Fowler
>Priority: Major
>
> If I run "mvn package" in the root Metron directory then compilation fails 
> because of a null reference in  
> {{./metron-platform/metron-common/src/test/java/org/apache/metron/common/error/MetronErrorTest.java}}.
>  This happens in {{getJSONObjectShouldReturnBasicInformation}} on line 56 
> because  {{errorJSON.get(Constants.ErrorFields.HOSTNAME.getName())}} is 
> assumed to return a string and the {{length()}} method is called on it. 
> Because the assert doesn't use a message then no obvious reason why it fails 
> is logged.
> The underlying problem is that 
> {{metron-platform/metron-common/src/main/java/org/apache/metron/common/error/MetronError.java}}
>  falls through to a {{catch}} block in {{addHostname()}} when 
> {{netAddress.getLocalHost().getHostName()}} exceptions with {{Name or service 
> not known}} for the host name.
> Setting a hostname that resolves is a build requirement that is out of 
> Metron's control, but if the code specifically handles the fact that it won't 
> always be retrieved then it seems problematic to have tests that assume it 
> works _and_ not make it clear what the failure is when it occurs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1299) MetronError tests fail if hostname isn't set

2018-02-02 Thread Otto Fowler (JIRA) 

 [ 
https://issues.apache.org/jira/browse/METRON-1299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler reassigned METRON-1299:
---

Assignee: Otto Fowler  (was: Mohan)

> MetronError tests fail if hostname isn't set
> 
>
> Key: METRON-1299
> URL: https://issues.apache.org/jira/browse/METRON-1299
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.1
> Environment: openSUSE Tumbleweed 20171102, OpenJDK 1.8.0_144
>Reporter: Stuart Bertram
>Assignee: Otto Fowler
>Priority: Major
>
> If I run "mvn package" in the root Metron directory then compilation fails 
> because of a null reference in  
> {{./metron-platform/metron-common/src/test/java/org/apache/metron/common/error/MetronErrorTest.java}}.
>  This happens in {{getJSONObjectShouldReturnBasicInformation}} on line 56 
> because  {{errorJSON.get(Constants.ErrorFields.HOSTNAME.getName())}} is 
> assumed to return a string and the {{length()}} method is called on it. 
> Because the assert doesn't use a message then no obvious reason why it fails 
> is logged.
> The underlying problem is that 
> {{metron-platform/metron-common/src/main/java/org/apache/metron/common/error/MetronError.java}}
>  falls through to a {{catch}} block in {{addHostname()}} when 
> {{netAddress.getLocalHost().getHostName()}} exceptions with {{Name or service 
> not known}} for the host name.
> Setting a hostname that resolves is a build requirement that is out of 
> Metron's control, but if the code specifically handles the fact that it won't 
> always be retrieved then it seems problematic to have tests that assume it 
> works _and_ not make it clear what the failure is when it occurs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1441) Create complementary Solr schemas for the main sensors

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350983#comment-16350983
 ] 

ASF GitHub Bot commented on METRON-1441:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/922
  
Ok, I did the following:
* Augmented the readme to point to the Solr documentation around schemas.  
Keep in mind, this is intermediate work that will feed into the "install Solr" 
work
* Added yaf and error schemas
* Renamed the test to an integration test
* Moved the data from multiline to separate files


> Create complementary Solr schemas for the main sensors
> --
>
> Key: METRON-1441
> URL: https://issues.apache.org/jira/browse/METRON-1441
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> We have ES templates for bro, snort, yaf, and error, we need corresponding 
> solr schemas for these collections.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1230) As a stopgap prior to METRON-777, add more simplistic sideloading of custom Parsers

2018-02-02 Thread Otto Fowler (JIRA) 

 [ 
https://issues.apache.org/jira/browse/METRON-1230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler reassigned METRON-1230:
---

Assignee: Casey Stella

> As a stopgap prior to METRON-777, add more simplistic sideloading of custom 
> Parsers
> ---
>
> Key: METRON-1230
> URL: https://issues.apache.org/jira/browse/METRON-1230
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
>Priority: Major
>
> Until we get 777 in, it'd be nice to have a simple ability using the normal 
> storm functionality to enable users to provide custom parsers without forking 
> Metron.  This should be done via simply creating a jar with their code (and 
> bundled dependencies) and have it picked up and available to the REST Service 
> (and consequently the management UI) as well as the start_parser_topology.sh 
> command.
> This should be minimal movement as we have a more robust solution coming with 
> METRON-777.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1302) Split up Indexing Topology into batch and random access sections

2018-02-02 Thread Otto Fowler (JIRA) 

 [ 
https://issues.apache.org/jira/browse/METRON-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otto Fowler reassigned METRON-1302:
---

Assignee: Casey Stella

> Split up Indexing Topology into batch and random access sections
> 
>
> Key: METRON-1302
> URL: https://issues.apache.org/jira/browse/METRON-1302
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
>Priority: Major
>
> Currently we have the indexing topology handle writing to both random access 
> indices (e.g. elasticsearch) as well as batch write indices (e.g. hdfs).  We 
> should split these up and configure them separately.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-809) Error while running metron_full_install.yml

2018-02-02 Thread Otto Fowler (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350975#comment-16350975
 ] 

Otto Fowler commented on METRON-809:


Closing inactive

> Error while running metron_full_install.yml
> ---
>
> Key: METRON-809
> URL: https://issues.apache.org/jira/browse/METRON-809
> Project: Metron
>  Issue Type: Bug
> Environment: Centos 6
>Reporter: shriram
>Priority: Blocker
> Attachments: Capture.JPG
>
>
> I am trying to install Metron cluster in AWS with CentOS 6. I encountered the 
> following error while running metron_full_install.yml, at its last step:
> "[ERROR] Failed to execute goal 
> org.codehaus.mojo:exec-maven-plugin:1.5.0:exec (docker-build) on project 
> metron-rpm: Command execution failed. Cannot run program \"docker\" (in 
> directory 
> \"/opt/incubator-metron/metron-deployment/packaging/docker/rpm-docker\"): 
> error=2, No such file or directory -> [Help 1]", "[ERROR] ", "[ERROR] To see 
> the full stack trace of the errors, re-run Maven with the -e switch.", 
> "[ERROR] Re-run Maven using the -X switch to enable full debug logging.", 
> "[ERROR] ", "[ERROR] For more information about the errors and possible 
> solutions, please read the following articles:", "[ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException;, 
> "[ERROR] ", "[ERROR] After correcting the problems, you can resume the build 
> with the command", "[ERROR]   mvn  -rf :metron-rpm"], "warnings": []}
> Attaching screenshot for more details.
> Thanks.
> Regards,
> Shriram



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1437) STELLAR: New Shell function to make a list from some number of variables

2018-02-02 Thread Otto Fowler (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350973#comment-16350973
 ] 

Otto Fowler commented on METRON-1437:
-

I'm not to bright.  "[var, var2, var4]"

 

> STELLAR: New Shell function to make a list from some number of variables
> 
>
> Key: METRON-1437
> URL: https://issues.apache.org/jira/browse/METRON-1437
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> The ability in the shell to make a list of a group of variables, the idea 
> being you can then pass that list to other functions



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1212) Bundles and Maven Plugin

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350940#comment-16350940
 ] 

ASF GitHub Bot commented on METRON-1212:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/865
  
Hey @JonZeolla how is it going?



> Bundles and Maven Plugin
> 
>
> Key: METRON-1212
> URL: https://issues.apache.org/jira/browse/METRON-1212
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>  Labels: metron-feature-canidate, 
> metron-feature-extensions-parsers
>
> The first effort will be to land the bundle system and supporting maven 
> plugin on master



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1442) Rest endpoints for operations on Indexing topology is broken after splitting the indexing topology

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350926#comment-16350926
 ] 

ASF GitHub Bot commented on METRON-1442:


GitHub user MohanDV opened a pull request:

https://github.com/apache/metron/pull/923

METRON-1442: Split rest end points for indexing topology into random access 
indexing and batch indexing


## Contributor Comments
Split rest end points for indexing topology into random access indexing 
topology and batch indexing topology to support the 
start/stop/activate/deactivate/status operations on the respective topologies.
**Steps to Verify**
1 . Spin up Full Dev
2. Go to Swagger at http://node1:8082/swagger-ui.html#!/storm-controller/
3. You should see the below newly added rest endpoints
https://user-images.githubusercontent.com/12934693/35753803-1f96b6c4-0887-11e8-9d55-6e185e34cc52.png;>


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MohanDV/metron METRON-1442

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/923.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #923


commit 59a5f26195032c33e0cf56a0abf0153cda09d71d
Author: Mohan Venkateshaiah 
Date:   2018-02-02T20:24:34Z

Splitted rest end points for indexing topology to support the 
start/stop/activate/deactivate/status operations on the randomaccess and batch 
indexing topology




> Rest endpoints for operations on Indexing topology is broken after splitting 
> the indexing topology
> --
>
> Key: METRON-1442
> URL: https://issues.apache.org/jira/browse/METRON-1442
> Project: Metron
>  Issue Type: Bug
>Reporter: Mohan
>Assignee: Mohan
>Priority: Major
>
> Rest endpoints for operations on Indexing topology is broken after splitting 
> the

[jira] [Commented] (METRON-1441) Create complementary Solr schemas for the main sensors

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350714#comment-16350714
 ] 

ASF GitHub Bot commented on METRON-1441:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/922#discussion_r165711148
  
--- Diff: 
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/schema/SchemaTranslatorTest.java
 ---
@@ -0,0 +1,188 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.schema;
+
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.integration.UnableToStartException;
+import org.apache.metron.solr.integration.components.SolrComponent;
+import org.apache.metron.solr.writer.SolrWriter;
+import org.json.simple.JSONObject;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.util.*;
+
+public class SchemaTranslatorTest {
+
+  /**

+{"adapter.threatinteladapter.end.ts":"1517499201357","bro_timestamp":"1517499194.7338","ip_dst_port":8080,"enrichmentsplitterbolt.splitter.end.ts":"1517499201202","enrichmentsplitterbolt.splitter.begin.ts":"1517499201200","adapter.hostfromjsonlistadapter.end.ts":"1517499201207","adapter.geoadapter.begin.ts":"1517499201209","uid":"CUrRne3iLIxXavQtci","trans_depth":143,"protocol":"http","original_string":"HTTP
 | id.orig_p:50451 method:GET request_body_len:0 id.resp_p:8080 
uri:\/api\/v1\/clusters\/metron_cluster\/services\/KAFKA\/components\/KAFKA_BROKER?fields=metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsBytesInPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsBytesOutPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsMessagesInPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/controller\/KafkaController\/ActiveControllerCount[1484165330,1484168930,15],metrics\/kafka\/controller\/ControllerStats\/LeaderElectionRateAndTimeMs\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/controller\/ControllerStats\/UncleanLeaderElectionsPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaFetcherManager\/Replica-MaxLag[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaManager\/PartitionCount[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaManager\/UnderReplicatedPartitions[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaManager\/LeaderCount[1484165330,1484168930,15]=null_padding&_=1484168930776
 tags:[] uid:CUrRne3iLIxXavQtci referrer:http:\/\/node1:8080\/ trans_depth:143 
host:node1 id.orig_h:192.168.66.1 response_body_len:0 user_agent:Mozilla\/5.0 
(Macintosh; Intel Mac OS X 10_12_2) AppleWebKit\/537.36 (KHTML, like Gecko) 
Chrome\/55.0.2883.95 Safari\/537.36 ts:1517499194.7338 
id.resp_h:192.168.66.121","ip_dst_addr":"192.168.66.121","threatinteljoinbolt.joiner.ts":"1517499201359","host":"node1","enrichmentjoinbolt.joiner.ts":"1517499201212","adapter.hostfromjsonlistadapter.begin.ts":"1517499201206","threatintelsplitterbolt.splitter.begin.ts":"1517499201215","ip_src_addr":"192.168.66.1","user_agent":"Mozilla\/5.0
 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit\/537.36 (KHTML, like Gecko) 
Chrome\/55.0.2883.95

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350575#comment-16350575
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/920


> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1436) Manually Install Solr Cloud in Full Dev

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350547#comment-16350547
 ] 

ASF GitHub Bot commented on METRON-1436:


Github user mmiklavc closed the pull request at:

https://github.com/apache/metron/pull/918


> Manually Install Solr Cloud in Full Dev
> ---
>
> Key: METRON-1436
> URL: https://issues.apache.org/jira/browse/METRON-1436
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
> Script and documentation for getting Solr setup in full dev



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350434#comment-16350434
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165663581
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/ShellFunctionsTest.java
 ---
@@ -40,8 +45,8 @@
   );
 
   Context context = new Context.Builder()
-.with(Context.Capabilities.SHELL_VARIABLES , () -> variables)
-.build();
+.with(Context.Capabilities.SHELL_VARIABLES , () -> 
variables).build();
--- End diff --

Right now, I never let my IDE reformat for me.  Like you said, if we get 
the code base matching check style and I can load that style into my IDE, then 
I'd gladly let it do most of the work for me.

Maybe I'll open a discuss thread.  I don't know how to handle this kind of 
thing and it happens all the time.

But for this specific scenario in your PR, it really doesn't matter either 
way.  I think you're good to go either way.


> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350435#comment-16350435
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/920
  
+1 Thanks @ottobackwards 


> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1441) Create complementary Solr schemas for the main sensors

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350427#comment-16350427
 ] 

ASF GitHub Bot commented on METRON-1441:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/922#discussion_r165662614
  
--- Diff: 
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/schema/SchemaTranslatorTest.java
 ---
@@ -0,0 +1,188 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.schema;
+
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.integration.UnableToStartException;
+import org.apache.metron.solr.integration.components.SolrComponent;
+import org.apache.metron.solr.writer.SolrWriter;
+import org.json.simple.JSONObject;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.util.*;
+
+public class SchemaTranslatorTest {
+
+  /**

+{"adapter.threatinteladapter.end.ts":"1517499201357","bro_timestamp":"1517499194.7338","ip_dst_port":8080,"enrichmentsplitterbolt.splitter.end.ts":"1517499201202","enrichmentsplitterbolt.splitter.begin.ts":"1517499201200","adapter.hostfromjsonlistadapter.end.ts":"1517499201207","adapter.geoadapter.begin.ts":"1517499201209","uid":"CUrRne3iLIxXavQtci","trans_depth":143,"protocol":"http","original_string":"HTTP
 | id.orig_p:50451 method:GET request_body_len:0 id.resp_p:8080 
uri:\/api\/v1\/clusters\/metron_cluster\/services\/KAFKA\/components\/KAFKA_BROKER?fields=metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsBytesInPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsBytesOutPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsMessagesInPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/controller\/KafkaController\/ActiveControllerCount[1484165330,1484168930,15],metrics\/kafka\/controller\/ControllerStats\/LeaderElectionRateAndTimeMs\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/controller\/ControllerStats\/UncleanLeaderElectionsPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaFetcherManager\/Replica-MaxLag[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaManager\/PartitionCount[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaManager\/UnderReplicatedPartitions[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaManager\/LeaderCount[1484165330,1484168930,15]=null_padding&_=1484168930776
 tags:[] uid:CUrRne3iLIxXavQtci referrer:http:\/\/node1:8080\/ trans_depth:143 
host:node1 id.orig_h:192.168.66.1 response_body_len:0 user_agent:Mozilla\/5.0 
(Macintosh; Intel Mac OS X 10_12_2) AppleWebKit\/537.36 (KHTML, like Gecko) 
Chrome\/55.0.2883.95 Safari\/537.36 ts:1517499194.7338 
id.resp_h:192.168.66.121","ip_dst_addr":"192.168.66.121","threatinteljoinbolt.joiner.ts":"1517499201359","host":"node1","enrichmentjoinbolt.joiner.ts":"1517499201212","adapter.hostfromjsonlistadapter.begin.ts":"1517499201206","threatintelsplitterbolt.splitter.begin.ts":"1517499201215","ip_src_addr":"192.168.66.1","user_agent":"Mozilla\/5.0
 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit\/537.36 (KHTML, like Gecko) 
Chrome\/55.0.2883.95

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350409#comment-16350409
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165660358
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/cli/PausableInput.java
 ---
@@ -36,8 +37,8 @@
  *
  */
 public class PausableInput extends InputStream {
-  InputStream in = System.in;
-  boolean paused = false;
+  private InputStream in = System.in;
+  private AtomicBoolean paused = new AtomicBoolean(false);
--- End diff --

Good find!  I'm sure that was frustrating to dig into.


> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350422#comment-16350422
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165662017
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/ShellFunctionsTest.java
 ---
@@ -40,8 +45,8 @@
   );
 
   Context context = new Context.Builder()
-.with(Context.Capabilities.SHELL_VARIABLES , () -> variables)
-.build();
+.with(Context.Capabilities.SHELL_VARIABLES , () -> 
variables).build();
--- End diff --

How do you have your formatting preferences set to get the above?


> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350419#comment-16350419
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165661676
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/ShellFunctionsTest.java
 ---
@@ -40,8 +45,8 @@
   );
 
   Context context = new Context.Builder()
-.with(Context.Capabilities.SHELL_VARIABLES , () -> variables)
-.build();
+.with(Context.Capabilities.SHELL_VARIABLES , () -> 
variables).build();
--- End diff --

I didn't reformat that like that on purpose.  I had a period where I was 
setting the CONSOLE capability.  When I removed it, it just worked out like 
this.

Even if I select and format in intellij it doesn't change the .build() to 
the next line.

I think what you have above is fine.  I would like it to 'just' happen if I 
format code though, since it is easy for things to slip through.

It is tough right now, because so much of the codebase isn't formatted to 
check style, and I don't think we want every pr to include a lot of formatting 
changes.



> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1435) Management UI cannot save json objects in advanced config

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350413#comment-16350413
 ] 

ASF GitHub Bot commented on METRON-1435:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/917


> Management UI cannot save json objects in advanced config
> -
>
> Key: METRON-1435
> URL: https://issues.apache.org/jira/browse/METRON-1435
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> Currently any value entered into the Advanced Config section of the 
> Management UI is converted to string.  It should allow arrays or json objects 
> to be saved as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350404#comment-16350404
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165659854
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/cli/PausableInput.java
 ---
@@ -36,8 +37,8 @@
  *
  */
 public class PausableInput extends InputStream {
-  InputStream in = System.in;
-  boolean paused = false;
+  private InputStream in = System.in;
+  private AtomicBoolean paused = new AtomicBoolean(false);
--- End diff --

I found in travis and locally, that the PausableInput was hanging, my 
builds where not completing.  PausableInput.INSTANCE.unpause(); was never 
returning.

I couldn't figure out why it was locked.  Reading the class documentation 
and the trying to understand the threading in the shell, I looked at the class 
and saw some inconsistencies with how and when we checked if we where paused in 
the different read() calls, and also that the flag field was not volatile or 
atomic.  I refactored this things to what I *think* would be the correct and 
consistent approaches and  it resolved my issue.


> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1441) Create complementary Solr schemas for the main sensors

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350386#comment-16350386
 ] 

ASF GitHub Bot commented on METRON-1441:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/922#discussion_r165656511
  
--- Diff: 
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/schema/SchemaTranslatorTest.java
 ---
@@ -0,0 +1,188 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.schema;
+
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.integration.UnableToStartException;
+import org.apache.metron.solr.integration.components.SolrComponent;
+import org.apache.metron.solr.writer.SolrWriter;
+import org.json.simple.JSONObject;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.util.*;
+
--- End diff --

Yes, I absolutely can.


> Create complementary Solr schemas for the main sensors
> --
>
> Key: METRON-1441
> URL: https://issues.apache.org/jira/browse/METRON-1441
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> We have ES templates for bro, snort, yaf, and error, we need corresponding 
> solr schemas for these collections.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1436) Manually Install Solr Cloud in Full Dev

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350393#comment-16350393
 ] 

ASF GitHub Bot commented on METRON-1436:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/918
  
Thanks for the updates.  I'm +1 on including this in the feature branch.


> Manually Install Solr Cloud in Full Dev
> ---
>
> Key: METRON-1436
> URL: https://issues.apache.org/jira/browse/METRON-1436
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
> Script and documentation for getting Solr setup in full dev



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1441) Create complementary Solr schemas for the main sensors

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350392#comment-16350392
 ] 

ASF GitHub Bot commented on METRON-1441:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/922
  
@ottobackwards Very likely these schema files won't stay in this spot, but 
the final resting spot won't be apparent until we figure out how to 
automatically apply the schemas.  Treat this PR as just unlocking progress for 
downstream PRs (like correcting SolrWriter to write to Solr again).


> Create complementary Solr schemas for the main sensors
> --
>
> Key: METRON-1441
> URL: https://issues.apache.org/jira/browse/METRON-1441
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> We have ES templates for bro, snort, yaf, and error, we need corresponding 
> solr schemas for these collections.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1441) Create complementary Solr schemas for the main sensors

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350387#comment-16350387
 ] 

ASF GitHub Bot commented on METRON-1441:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/922#discussion_r165656636
  
--- Diff: 
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/schema/SchemaTranslatorTest.java
 ---
@@ -0,0 +1,188 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.schema;
+
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.integration.UnableToStartException;
+import org.apache.metron.solr.integration.components.SolrComponent;
+import org.apache.metron.solr.writer.SolrWriter;
+import org.json.simple.JSONObject;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.util.*;
+
+public class SchemaTranslatorTest {
+
+  /**

+{"adapter.threatinteladapter.end.ts":"1517499201357","bro_timestamp":"1517499194.7338","ip_dst_port":8080,"enrichmentsplitterbolt.splitter.end.ts":"1517499201202","enrichmentsplitterbolt.splitter.begin.ts":"1517499201200","adapter.hostfromjsonlistadapter.end.ts":"1517499201207","adapter.geoadapter.begin.ts":"1517499201209","uid":"CUrRne3iLIxXavQtci","trans_depth":143,"protocol":"http","original_string":"HTTP
 | id.orig_p:50451 method:GET request_body_len:0 id.resp_p:8080 
uri:\/api\/v1\/clusters\/metron_cluster\/services\/KAFKA\/components\/KAFKA_BROKER?fields=metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsBytesInPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsBytesOutPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/server\/BrokerTopicMetrics\/AllTopicsMessagesInPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/controller\/KafkaController\/ActiveControllerCount[1484165330,1484168930,15],metrics\/kafka\/controller\/ControllerStats\/LeaderElectionRateAndTimeMs\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/controller\/ControllerStats\/UncleanLeaderElectionsPerSec\/1MinuteRate[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaFetcherManager\/Replica-MaxLag[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaManager\/PartitionCount[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaManager\/UnderReplicatedPartitions[1484165330,1484168930,15],metrics\/kafka\/server\/ReplicaManager\/LeaderCount[1484165330,1484168930,15]=null_padding&_=1484168930776
 tags:[] uid:CUrRne3iLIxXavQtci referrer:http:\/\/node1:8080\/ trans_depth:143 
host:node1 id.orig_h:192.168.66.1 response_body_len:0 user_agent:Mozilla\/5.0 
(Macintosh; Intel Mac OS X 10_12_2) AppleWebKit\/537.36 (KHTML, like Gecko) 
Chrome\/55.0.2883.95 Safari\/537.36 ts:1517499194.7338 
id.resp_h:192.168.66.121","ip_dst_addr":"192.168.66.121","threatinteljoinbolt.joiner.ts":"1517499201359","host":"node1","enrichmentjoinbolt.joiner.ts":"1517499201212","adapter.hostfromjsonlistadapter.begin.ts":"1517499201206","threatintelsplitterbolt.splitter.begin.ts":"1517499201215","ip_src_addr":"192.168.66.1","user_agent":"Mozilla\/5.0
 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit\/537.36 (KHTML, like Gecko) 
Chrome\/55.0.2883.95

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350380#comment-16350380
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165654991
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/cli/PausableInput.java
 ---
@@ -36,8 +37,8 @@
  *
  */
 public class PausableInput extends InputStream {
-  InputStream in = System.in;
-  boolean paused = false;
+  private InputStream in = System.in;
+  private AtomicBoolean paused = new AtomicBoolean(false);
--- End diff --

What problem were you solving here @ottobackwards ?  Is this bit access by 
multiple threads?


> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

2018-02-02 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350379#comment-16350379
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165653517
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/ShellFunctionsTest.java
 ---
@@ -40,8 +45,8 @@
   );
 
   Context context = new Context.Builder()
-.with(Context.Capabilities.SHELL_VARIABLES , () -> variables)
-.build();
+.with(Context.Capabilities.SHELL_VARIABLES , () -> 
variables).build();
--- End diff --

I am actually interested in what direction as a project we should be taking 
with these types of fluent, chained statements.  I run across this all the time 
and I want to know the 'right' way that I should be doing it for the project.

IMHO, the way it was (separated by a line break) is more readable.  
Meaning, a long set of chained statements should be separated by line breaks.  
For example...
```
  result = new ProfileMeasurement()
  .withProfileName(profileName)
  .withEntity(entity)
  .withGroups(groups)
  .withPeriod(period)
  .withProfileValue(profileValue)
  .withTriageValues(triageValues)
  .withDefinition(definition);
```

But, of course, in terms of code style my opinion doesn't matter.  It is 
all about our style guidelines. 
 What does the Google code style guidelines say?   

Doesn't 
[this](https://google.github.io/styleguide/javaguide.html#s4.5.1-line-wrapping-where-to-break)
 support what I have said above about line breaks in this case?  




> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1442) Rest endpoints for operations on Indexing topology is broken after splitting the indexing topology

2018-02-02 Thread Mohan (JIRA) 
Mohan created METRON-1442:
-

 Summary: Rest endpoints for operations on Indexing topology is 
broken after splitting the indexing topology
 Key: METRON-1442
 URL: https://issues.apache.org/jira/browse/METRON-1442
 Project: Metron
  Issue Type: Bug
Reporter: Mohan
Assignee: Mohan


Rest endpoints for operations on Indexing topology is broken after splitting 
the indexing topology into random_access_indexing and batch_indexing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)