Re: [DISCUSS] Quick Poll - How You are Using & Contributing to Metron? - 2018-01-30

[Ahmed Shah]

Hello Metron developers and users;


Here are the results of the survey so far.
---
Where respondents are located:
Canada, USA

Types of data collected:
Various Logs, Honeypot Data, Firewall, IDS, Server Data

Kind of sensors used:
JSONMap, Syslog, Bro, Suricata, Dionaea, Netflow

How are you using Metron?:
Evaluation, SIEM, As SOC front end

How are you CURRENTLY contributing?:
None/Not at present, Documentation, Demos, Pull Requests

In what areas would you LIKE to contribute to Metron?:
Feedback/Suggestions (80% of  respondents)
Supporting others (20% of  respondents)
Documentation (40% of  respondents)
Cloud Deployment (20% of  respondents)
Dashboards (60% of  respondents)
Adding Sensors (40% of  respondents)
Machine Learning (40% of  respondents)
Build Operations (20% of  respondents)
Adoption Strategy  (40% of  respondents)
General Coding  (20% of  respondents)
Mentoring  (20% of  respondents)


Anything else you would like to mention?

like to see stability in install process

---


The survey is still open if you still wish to share your thoughts and interest 
in contributing to the community:

https://docs.google.com/forms/d/e/1FAIpQLSfSNwMTiY8FS4ayjORYNg4qWy_3WPuM0uL_O7Mn6IptwcIW4w/viewform?usp=sf_link




-Ahmed
___
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com



From: Ahmed Shah
Sent: January 30, 2018 8:02 AM
To: dev@metron.apache.org; u...@metron.apache.org
Subject: [DISCUSS] Quick Poll - How You are Using & Contributing to Metron? - 
2018-01-30


Hello Metron developers and users;

The following questionnaire will help gather information on how the community 
is using Metron. It will also help understand which areas people are most 
interested in contributing. The questionnaire shouldn't take long to complete 
(less then 10 minutes) and many of the fields are optional. If you fill in this 
questionnaire you have contributed 

https://docs.google.com/forms/d/e/1FAIpQLSfSNwMTiY8FS4ayjORYNg4qWy_3WPuM0uL_O7Mn6IptwcIW4w/viewform?usp=sf_link

[https://lh6.googleusercontent.com/jZgsb4uggh5jrOBGbfyQnO_rBW8siTRx1u3Pe0_mltuH8zz5HfxvXO97JaMvdmSH8u0=w1200-h630-p]

[DISCUSS] Quick Poll - How You are Using & Contributing to Metron? - 
2018-01-30
docs.google.com
Please complete before Jan 31st(Wed) - 2018, 16:00 UTC.





Information collected today will be shared at tomorrows user group meeting.


-Ahmed
___
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com

[GitHub] metron issue #918: METRON-1436: Manually Install Solr Cloud in Full Dev

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/918
  
I tested it out and worked fine.  I think it's a good start.  +1 as long as 
other commenters are satisfied.


---

[GitHub] metron pull request #918: METRON-1436: Manually Install Solr Cloud in Full D...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/918#discussion_r165192051
  
--- Diff: metron-platform/metron-solr/README.md ---
@@ -0,0 +1,52 @@
+
+# Solr in Metron
+
+## Table of Contents
+
+* [Introduction](#introduction)
+* [Installing](#installing)
+
+## Introduction
+
+Metron ships with Solr 6.6.2. Solr Cloud can be used as the real-time 
portion of the datastore resulting from 
[metron-indexing](../metron-indexing/README.md).
--- End diff --

Can you make this "... Solr 6.6.2 support", so it's obvious we're not 
actually including Solr?

I'd also add the blurb / link from the script about production use here as 
well.


---

[GitHub] metron issue #916: METRON-1434 - Ability to deploy Metron full dev as a sing...

[as22323]

Github user as22323 commented on the issue:

https://github.com/apache/metron/pull/916
  
I have taken down the AMI (ami-93cb4ff7) from the AWS Community 
marketplace. The AMI id was listed in the README.md. The AMI was originally 
created to be a test/proof-of-concept for launching Metron in AWS.


---

Re: When things change in hdfs, how do we know

[Otto Fowler]

It depends if the transaction id is the same per instance.  If it is, then
we can de-doupe as long as we put the tx id into the event json.



On January 31, 2018 at 12:40:15, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

I take it your service would just be a thin daemon along the lines of the
PoC you linked, which makes a lot of sense, delegating the actual
notification to the zookeeper bits we already have.

That makes sense to me. One other question would be around the availability
of that service (which is not exactly critical, but would be nice to be
able to run HA). As far as I can see it’s not likely to be stateful, and as
long as there is some sort of de-dupe you could have two or more running.
Is that worth chewing, or do we just need one running and accept occasional
outages of the rarely firing non-critical service?

Simon

> On 31 Jan 2018, at 17:24, Otto Fowler  wrote:
>
> No,
>
> I would propose a new Ambari Service, the did the notify->zookeeper
stuff.
> Did you not see my awesome ascii art diagram?
>
>
>
>
> On January 31, 2018 at 11:51:51, Casey Stella (ceste...@gmail.com) wrote:
>
> Well, it'll be one listener per worker and if you have a lot of workers,
> it's going to be a bad time probably.
>
> On Wed, Jan 31, 2018 at 11:50 AM, Otto Fowler 
> wrote:
>
>> I don’t think the Unstable means the implementation will crash. I think
>> it means
>> it is a newish-api, and there should be 1 listeners.
>>
>> Having 1 listener shouldn’t be an issue.
>>
>>
>>
>> On January 31, 2018 at 11:45:54, Casey Stella (ceste...@gmail.com)
wrote:
>>
>> Hmm, I have heard this feedback before. Perhaps a more low-key approach
>> would be either a static timer that checked or a timer bolt that sent a
>> periodic timer and the parser bolt reconfigured the parser (or indeed we
>> added a Reloadable interface with a 'reload' method). We could be smart
>> also and only set up the topology with the timer bolt if the parser
>> actually implemented the Reloadable interface. Just some thoughts that
>> might be easy and avoid instability.
>>
>> On Tue, Jan 30, 2018 at 3:42 PM, Otto Fowler 
>> wrote:
>>
>>> It is still @unstable, but the jiras :
>>> https://issues.apache.org/jira/browse/HDFS-8940?jql=
>>> project%20%3D%20HDFS%20AND%20status%20in%20(Open%2C%20%
>>> 22In%20Progress%22)%20AND%20text%20~%20%22INotify%22
>>> that I see are stall from over the summer.
>>>
>>> They also seem geared to scale or changing the filter object not the
api.
>>>
>>>
>>>
>>> On January 30, 2018 at 14:19:56, JJ Meyer (jjmey...@gmail.com) wrote:
>>>
>>> Hello all,
>>>
>>> I had created a NiFi processor a long time back that used the inotify
>> API.
>>> One thing I noticed while working with it is that it is marked with the
>>> `Unstable` annotation. It may be worth checking if anymore work is
going
>> on
>>> with it and if it will impact this (if it hasn't already been looked
>> into).
>>>
>>> Thanks,
>>> JJ
>>>
>>> On Mon, Jan 29, 2018 at 7:27 AM, Otto Fowler 
>>> wrote:
>>>
 I have updated the jira as well


 On January 29, 2018 at 08:22:34, Otto Fowler (ottobackwa...@gmail.com)
 wrote:

 https://github.com/ottobackwards/hdfs-inotify-zookeeper

>>>
>>
>>

[GitHub] metron issue #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/911
  
I believe all feedback has been addressed.  The ElasticsearchDao refactor 
was not trivial and was more complex than the SolrDao.  There were more 
dependencies between the various Daos so it's not as simple as I would like.  
Let me know what you think.


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165142091
  
--- Diff: 
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/integration/components/SolrComponent.java
 ---
@@ -158,4 +162,16 @@ public boolean hasCollection(String collection) {
 }
 return docs;
   }
+
+  public void addDocs(String collection, List> docs)
+  throws IOException, SolrServerException {
+CloudSolrClient solr = miniSolrCloudCluster.getSolrClient();
+solr.setDefaultCollection(collection);
+Collection solrInputDocuments = 
docs.stream().map(doc -> {
+  SolrInputDocument solrInputDocument = new SolrInputDocument();
+  doc.entrySet().forEach(entry -> 
solrInputDocument.addField(entry.getKey(), entry.getValue()));
--- End diff --

Done


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165141979
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.Group;
+import org.apache.metron.indexing.dao.search.GroupOrder;
+import org.apache.metron.indexing.dao.search.GroupOrderType;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.GroupResult;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchDao;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.search.SearchResult;
+import org.apache.metron.indexing.dao.search.SortField;
+import org.apache.metron.indexing.dao.search.SortOrder;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.client.solrj.response.FacetField.Count;
+import org.apache.solr.client.solrj.response.PivotField;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrSearchDao implements SearchDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+  private AccessConfig accessConfig;
+
+  public SolrSearchDao(SolrClient client, AccessConfig accessConfig) {
+this.client = client;
+this.accessConfig = accessConfig;
+  }
+
+  @Override
+  public SearchResponse search(SearchRequest searchRequest) throws 
InvalidSearchException {
+if (searchRequest.getQuery() == null) {
+  throw new InvalidSearchException("Search query is invalid: null");
+}
+if (client == null) {
+  throw new InvalidSearchException("Uninitialized Dao!  You must call 
init() prior to use.");
+}
+if (searchRequest.getSize() > accessConfig.getMaxSearchResults()) {
+  throw new InvalidSearchException(
+  "Search result size must be less than " + 
accessConfig.getMaxSearchResults());
+}
+SolrQuery query = buildSearchRequest(searchRequest);
+try {
+  QueryResponse response = client.query(query);
+  return buildSearchResponse(searchRequest, response);
+} catch (IOException | SolrServerException e) {
+  String msg = e.getMessage();
+  LOG.error(msg, e);
+  throw new InvalidSearchException(msg, e);
+}
+  }
+
+  @Override
+  public GroupResponse group(GroupRequest groupRequest) throws 
InvalidSearchException {
+String groupNames = 
groupRequest.getGroups().stream().map(Grou

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165142013
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrUpdateDao.java
 ---
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Optional;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.metron.indexing.dao.update.UpdateDao;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrInputDocument;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrUpdateDao implements UpdateDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+
+  public SolrUpdateDao(SolrClient client) {
+this.client = client;
+  }
+
+  @Override
+  public void update(Document update, Optional index) throws 
IOException {
+try {
+  SolrInputDocument solrInputDocument = toSolrInputDocument(update);
+  if (index.isPresent()) {
+this.client.add(index.get(), solrInputDocument);
+  } else {
+this.client.add(solrInputDocument);
+  }
+} catch (SolrServerException e) {
+  throw new IOException(e);
+}
+  }
+
+  @Override
+  public void batchUpdate(Map> updates) throws 
IOException {
+// updates with a collection specified
+Map> solrCollectionUpdates = new 
HashMap<>();
+
+// updates with no collection specified
+Collection solrUpdates = new ArrayList<>();
+
+for(Entry> entry: updates.entrySet()) {
+  SolrInputDocument solrInputDocument = 
toSolrInputDocument(entry.getKey());
+  Optional index = entry.getValue();
+  if (index.isPresent()) {
+Collection solrInputDocuments = 
solrCollectionUpdates.get(index.get());
+if (solrInputDocuments == null) {
+  solrInputDocuments = new ArrayList<>();
+}
+solrInputDocuments.add(solrInputDocument);
+solrCollectionUpdates.put(index.get(), solrInputDocuments);
+  } else {
+solrUpdates.add(solrInputDocument);
+  }
+}
+try {
+  if (!solrCollectionUpdates.isEmpty()) {
+for(Entry> entry: 
solrCollectionUpdates.entrySet()) {
+  this.client.add(entry.getKey(), entry.getValue());
+}
+  } else {
+this.client.add(solrUpdates);
+  }
+} catch (SolrServerException e) {
+  throw new IOException(e);
+}
+  }
+
+  private SolrInputDocument toSolrInputDocument(Document document) {
+SolrInputDocument solrInputDocument = new SolrInputDocument();
+document.getDocument().entrySet().forEach(entry ->
--- End diff --

Done


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165142049
  
--- Diff: 
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/integration/SolrSearchIntegrationTest.java
 ---
@@ -0,0 +1,152 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.integration;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.SearchIntegrationTest;
+import org.apache.metron.indexing.dao.search.FieldType;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.integration.InMemoryComponent;
+import org.apache.metron.solr.dao.SolrDao;
+import org.apache.metron.solr.integration.components.SolrComponent;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.json.simple.JSONArray;
+import org.json.simple.parser.JSONParser;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class SolrSearchIntegrationTest extends SearchIntegrationTest {
+
+  private SolrComponent solrComponent;
+
+  @Override
+  protected IndexDao createDao() throws Exception {
+AccessConfig config = new AccessConfig();
+config.setMaxSearchResults(100);
+config.setMaxSearchGroups(100);
+config.setGlobalConfigSupplier( () ->
+new HashMap() {{
+  put("solr.zookeeper", solrComponent.getZookeeperUrl());
+}}
+);
+
+IndexDao dao = new SolrDao();
+dao.init(config);
+return dao;
+  }
+
+  @Override
+  protected InMemoryComponent startIndex() throws Exception {
+solrComponent = new SolrComponent.Builder()
+.addCollection("bro", 
"../metron-solr/src/test/resources/config/bro/conf")
+.addCollection("snort", 
"../metron-solr/src/test/resources/config/snort/conf")
+.build();
+solrComponent.start();
+return solrComponent;
+  }
+
+  @Override
--- End diff --

Done


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165141915
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.Group;
+import org.apache.metron.indexing.dao.search.GroupOrder;
+import org.apache.metron.indexing.dao.search.GroupOrderType;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.GroupResult;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchDao;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.search.SearchResult;
+import org.apache.metron.indexing.dao.search.SortField;
+import org.apache.metron.indexing.dao.search.SortOrder;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.client.solrj.response.FacetField.Count;
+import org.apache.solr.client.solrj.response.PivotField;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrSearchDao implements SearchDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+  private AccessConfig accessConfig;
+
+  public SolrSearchDao(SolrClient client, AccessConfig accessConfig) {
+this.client = client;
+this.accessConfig = accessConfig;
+  }
+
+  @Override
+  public SearchResponse search(SearchRequest searchRequest) throws 
InvalidSearchException {
+if (searchRequest.getQuery() == null) {
+  throw new InvalidSearchException("Search query is invalid: null");
+}
+if (client == null) {
+  throw new InvalidSearchException("Uninitialized Dao!  You must call 
init() prior to use.");
+}
+if (searchRequest.getSize() > accessConfig.getMaxSearchResults()) {
+  throw new InvalidSearchException(
+  "Search result size must be less than " + 
accessConfig.getMaxSearchResults());
+}
+SolrQuery query = buildSearchRequest(searchRequest);
+try {
+  QueryResponse response = client.query(query);
+  return buildSearchResponse(searchRequest, response);
+} catch (IOException | SolrServerException e) {
+  String msg = e.getMessage();
+  LOG.error(msg, e);
+  throw new InvalidSearchException(msg, e);
+}
+  }
+
+  @Override
+  public GroupResponse group(GroupRequest groupRequest) throws 
InvalidSearchException {
+String groupNames = 
groupRequest.getGroups().stream().map(Grou

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165141548
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.Group;
+import org.apache.metron.indexing.dao.search.GroupOrder;
+import org.apache.metron.indexing.dao.search.GroupOrderType;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.GroupResult;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchDao;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.search.SearchResult;
+import org.apache.metron.indexing.dao.search.SortField;
+import org.apache.metron.indexing.dao.search.SortOrder;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.client.solrj.response.FacetField.Count;
+import org.apache.solr.client.solrj.response.PivotField;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrSearchDao implements SearchDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+  private AccessConfig accessConfig;
+
+  public SolrSearchDao(SolrClient client, AccessConfig accessConfig) {
+this.client = client;
+this.accessConfig = accessConfig;
+  }
+
+  @Override
+  public SearchResponse search(SearchRequest searchRequest) throws 
InvalidSearchException {
+if (searchRequest.getQuery() == null) {
+  throw new InvalidSearchException("Search query is invalid: null");
+}
+if (client == null) {
+  throw new InvalidSearchException("Uninitialized Dao!  You must call 
init() prior to use.");
+}
+if (searchRequest.getSize() > accessConfig.getMaxSearchResults()) {
+  throw new InvalidSearchException(
+  "Search result size must be less than " + 
accessConfig.getMaxSearchResults());
+}
+SolrQuery query = buildSearchRequest(searchRequest);
+try {
+  QueryResponse response = client.query(query);
+  return buildSearchResponse(searchRequest, response);
+} catch (IOException | SolrServerException e) {
+  String msg = e.getMessage();
+  LOG.error(msg, e);
+  throw new InvalidSearchException(msg, e);
+}
+  }
+
+  @Override
+  public GroupResponse group(GroupRequest groupRequest) throws 
InvalidSearchException {
+String groupNames = 
groupRequest.getGroups().stream().map(Grou

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165141296
  
--- Diff: 
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java
 ---
@@ -655,83 +699,54 @@ public void disabled_facet_query_returns_null_count() 
throws Exception {
   }
 
   @Test
-  public void exceeding_max_resulsts_throws_exception() throws Exception {
+  public void missing_type_facet_query() throws Exception {
+SearchRequest request = JSONUtils.INSTANCE.load(missingTypeFacetQuery, 
SearchRequest.class);
+SearchResponse response = dao.search(request);
+Assert.assertEquals(10, response.getTotal());
+
+Map> facetCounts = response.getFacetCounts();
+Assert.assertEquals(1, facetCounts.size());
+Map snortFieldCounts = facetCounts.get("snort_field");
+Assert.assertEquals(5, snortFieldCounts.size());
+Assert.assertEquals(new Long(1), snortFieldCounts.get("50"));
--- End diff --

Done


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165141322
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrDao.java
 ---
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.ColumnMetadataDao;
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.search.FieldType;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrDao implements IndexDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+  public static final String ID_FIELD = "guid";
--- End diff --

Done


---

Re: When things change in hdfs, how do we know

[Simon Elliston Ball]

I take it your service would just be a thin daemon along the lines of the PoC 
you linked, which makes a lot of sense, delegating the actual notification to 
the zookeeper bits we already have.

That makes sense to me. One other question would be around the availability of 
that service (which is not exactly critical, but would be nice to be able to 
run HA). As far as I can see it’s not likely to be stateful, and as long as 
there is some sort of de-dupe you could have two or more running. Is that worth 
chewing, or do we just need one running and accept occasional outages of the 
rarely firing non-critical service?

Simon

> On 31 Jan 2018, at 17:24, Otto Fowler  wrote:
> 
> No,
> 
> I would propose a new Ambari Service, the did the notify->zookeeper stuff.
> Did you not see my awesome ascii art diagram?
> 
> 
> 
> 
> On January 31, 2018 at 11:51:51, Casey Stella (ceste...@gmail.com) wrote:
> 
> Well, it'll be one listener per worker and if you have a lot of workers,
> it's going to be a bad time probably.
> 
> On Wed, Jan 31, 2018 at 11:50 AM, Otto Fowler 
> wrote:
> 
>> I don’t think the Unstable means the implementation will crash.  I think
>> it means
>> it is a newish-api, and there should be 1 listeners.
>> 
>> Having 1 listener shouldn’t be an issue.
>> 
>> 
>> 
>> On January 31, 2018 at 11:45:54, Casey Stella (ceste...@gmail.com) wrote:
>> 
>> Hmm, I have heard this feedback before. Perhaps a more low-key approach
>> would be either a static timer that checked or a timer bolt that sent a
>> periodic timer and the parser bolt reconfigured the parser (or indeed we
>> added a Reloadable interface with a 'reload' method). We could be smart
>> also and only set up the topology with the timer bolt if the parser
>> actually implemented the Reloadable interface. Just some thoughts that
>> might be easy and avoid instability.
>> 
>> On Tue, Jan 30, 2018 at 3:42 PM, Otto Fowler 
>> wrote:
>> 
>>> It is still @unstable, but the jiras :
>>> https://issues.apache.org/jira/browse/HDFS-8940?jql=
>>> project%20%3D%20HDFS%20AND%20status%20in%20(Open%2C%20%
>>> 22In%20Progress%22)%20AND%20text%20~%20%22INotify%22
>>> that I see are stall from over the summer.
>>> 
>>> They also seem geared to scale or changing the filter object not the api.
>>> 
>>> 
>>> 
>>> On January 30, 2018 at 14:19:56, JJ Meyer (jjmey...@gmail.com) wrote:
>>> 
>>> Hello all,
>>> 
>>> I had created a NiFi processor a long time back that used the inotify
>> API.
>>> One thing I noticed while working with it is that it is marked with the
>>> `Unstable` annotation. It may be worth checking if anymore work is going
>> on
>>> with it and if it will impact this (if it hasn't already been looked
>> into).
>>> 
>>> Thanks,
>>> JJ
>>> 
>>> On Mon, Jan 29, 2018 at 7:27 AM, Otto Fowler 
>>> wrote:
>>> 
 I have updated the jira as well
 
 
 On January 29, 2018 at 08:22:34, Otto Fowler (ottobackwa...@gmail.com)
 wrote:
 
 https://github.com/ottobackwards/hdfs-inotify-zookeeper
 
>>> 
>> 
>> 

Re: When things change in hdfs, how do we know

[Otto Fowler]

No,

I would propose a new Ambari Service, the did the notify->zookeeper stuff.
Did you not see my awesome ascii art diagram?




On January 31, 2018 at 11:51:51, Casey Stella (ceste...@gmail.com) wrote:

Well, it'll be one listener per worker and if you have a lot of workers,
it's going to be a bad time probably.

On Wed, Jan 31, 2018 at 11:50 AM, Otto Fowler 
wrote:

> I don’t think the Unstable means the implementation will crash.  I think
> it means
> it is a newish-api, and there should be 1 listeners.
>
> Having 1 listener shouldn’t be an issue.
>
>
>
> On January 31, 2018 at 11:45:54, Casey Stella (ceste...@gmail.com) wrote:
>
> Hmm, I have heard this feedback before. Perhaps a more low-key approach
> would be either a static timer that checked or a timer bolt that sent a
> periodic timer and the parser bolt reconfigured the parser (or indeed we
> added a Reloadable interface with a 'reload' method). We could be smart
> also and only set up the topology with the timer bolt if the parser
> actually implemented the Reloadable interface. Just some thoughts that
> might be easy and avoid instability.
>
> On Tue, Jan 30, 2018 at 3:42 PM, Otto Fowler 
> wrote:
>
> > It is still @unstable, but the jiras :
> > https://issues.apache.org/jira/browse/HDFS-8940?jql=
> > project%20%3D%20HDFS%20AND%20status%20in%20(Open%2C%20%
> > 22In%20Progress%22)%20AND%20text%20~%20%22INotify%22
> > that I see are stall from over the summer.
> >
> > They also seem geared to scale or changing the filter object not the api.
> >
> >
> >
> > On January 30, 2018 at 14:19:56, JJ Meyer (jjmey...@gmail.com) wrote:
> >
> > Hello all,
> >
> > I had created a NiFi processor a long time back that used the inotify
> API.
> > One thing I noticed while working with it is that it is marked with the
> > `Unstable` annotation. It may be worth checking if anymore work is going
> on
> > with it and if it will impact this (if it hasn't already been looked
> into).
> >
> > Thanks,
> > JJ
> >
> > On Mon, Jan 29, 2018 at 7:27 AM, Otto Fowler 
> > wrote:
> >
> > > I have updated the jira as well
> > >
> > >
> > > On January 29, 2018 at 08:22:34, Otto Fowler (ottobackwa...@gmail.com)
> > > wrote:
> > >
> > > https://github.com/ottobackwards/hdfs-inotify-zookeeper
> > >
> >
>
>

Re: When things change in hdfs, how do we know

[Casey Stella]

Well, it'll be one listener per worker and if you have a lot of workers,
it's going to be a bad time probably.

On Wed, Jan 31, 2018 at 11:50 AM, Otto Fowler 
wrote:

> I don’t think the Unstable means the implementation will crash.  I think
> it means
> it is a newish-api, and there should be 1 listeners.
>
> Having 1 listener shouldn’t be an issue.
>
>
>
> On January 31, 2018 at 11:45:54, Casey Stella (ceste...@gmail.com) wrote:
>
> Hmm, I have heard this feedback before. Perhaps a more low-key approach
> would be either a static timer that checked or a timer bolt that sent a
> periodic timer and the parser bolt reconfigured the parser (or indeed we
> added a Reloadable interface with a 'reload' method). We could be smart
> also and only set up the topology with the timer bolt if the parser
> actually implemented the Reloadable interface. Just some thoughts that
> might be easy and avoid instability.
>
> On Tue, Jan 30, 2018 at 3:42 PM, Otto Fowler 
> wrote:
>
> > It is still @unstable, but the jiras :
> > https://issues.apache.org/jira/browse/HDFS-8940?jql=
> > project%20%3D%20HDFS%20AND%20status%20in%20(Open%2C%20%
> > 22In%20Progress%22)%20AND%20text%20~%20%22INotify%22
> > that I see are stall from over the summer.
> >
> > They also seem geared to scale or changing the filter object not the
> api.
> >
> >
> >
> > On January 30, 2018 at 14:19:56, JJ Meyer (jjmey...@gmail.com) wrote:
> >
> > Hello all,
> >
> > I had created a NiFi processor a long time back that used the inotify
> API.
> > One thing I noticed while working with it is that it is marked with the
> > `Unstable` annotation. It may be worth checking if anymore work is going
> on
> > with it and if it will impact this (if it hasn't already been looked
> into).
> >
> > Thanks,
> > JJ
> >
> > On Mon, Jan 29, 2018 at 7:27 AM, Otto Fowler 
> > wrote:
> >
> > > I have updated the jira as well
> > >
> > >
> > > On January 29, 2018 at 08:22:34, Otto Fowler (ottobackwa...@gmail.com)
>
> > > wrote:
> > >
> > > https://github.com/ottobackwards/hdfs-inotify-zookeeper
> > >
> >
>
>

Re: When things change in hdfs, how do we know

[Otto Fowler]

I don’t think the Unstable means the implementation will crash.  I think it
means
it is a newish-api, and there should be 1 listeners.

Having 1 listener shouldn’t be an issue.



On January 31, 2018 at 11:45:54, Casey Stella (ceste...@gmail.com) wrote:

Hmm, I have heard this feedback before. Perhaps a more low-key approach
would be either a static timer that checked or a timer bolt that sent a
periodic timer and the parser bolt reconfigured the parser (or indeed we
added a Reloadable interface with a 'reload' method). We could be smart
also and only set up the topology with the timer bolt if the parser
actually implemented the Reloadable interface. Just some thoughts that
might be easy and avoid instability.

On Tue, Jan 30, 2018 at 3:42 PM, Otto Fowler 
wrote:

> It is still @unstable, but the jiras :
> https://issues.apache.org/jira/browse/HDFS-8940?jql=
> project%20%3D%20HDFS%20AND%20status%20in%20(Open%2C%20%
> 22In%20Progress%22)%20AND%20text%20~%20%22INotify%22
> that I see are stall from over the summer.
>
> They also seem geared to scale or changing the filter object not the api.
>
>
>
> On January 30, 2018 at 14:19:56, JJ Meyer (jjmey...@gmail.com) wrote:
>
> Hello all,
>
> I had created a NiFi processor a long time back that used the inotify
API.
> One thing I noticed while working with it is that it is marked with the
> `Unstable` annotation. It may be worth checking if anymore work is going
on
> with it and if it will impact this (if it hasn't already been looked
into).
>
> Thanks,
> JJ
>
> On Mon, Jan 29, 2018 at 7:27 AM, Otto Fowler 
> wrote:
>
> > I have updated the jira as well
> >
> >
> > On January 29, 2018 at 08:22:34, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > https://github.com/ottobackwards/hdfs-inotify-zookeeper
> >
>

Re: When things change in hdfs, how do we know

[Casey Stella]

Hmm, I have heard this feedback before.  Perhaps a more low-key approach
would be either a static timer that checked or a timer bolt that sent a
periodic timer and the parser bolt reconfigured the parser (or indeed we
added a Reloadable interface with a 'reload' method).  We could be smart
also and only set up the topology with the timer bolt if the parser
actually implemented the Reloadable interface.  Just some thoughts that
might be easy and avoid instability.

On Tue, Jan 30, 2018 at 3:42 PM, Otto Fowler 
wrote:

> It is still @unstable, but the jiras :
> https://issues.apache.org/jira/browse/HDFS-8940?jql=
> project%20%3D%20HDFS%20AND%20status%20in%20(Open%2C%20%
> 22In%20Progress%22)%20AND%20text%20~%20%22INotify%22
> that I see are stall from over the summer.
>
> They also seem geared to scale or changing the filter object not the api.
>
>
>
> On January 30, 2018 at 14:19:56, JJ Meyer (jjmey...@gmail.com) wrote:
>
> Hello all,
>
> I had created a NiFi processor a long time back that used the inotify API.
> One thing I noticed while working with it is that it is marked with the
> `Unstable` annotation. It may be worth checking if anymore work is going on
> with it and if it will impact this (if it hasn't already been looked into).
>
> Thanks,
> JJ
>
> On Mon, Jan 29, 2018 at 7:27 AM, Otto Fowler 
> wrote:
>
> > I have updated the jira as well
> >
> >
> > On January 29, 2018 at 08:22:34, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > https://github.com/ottobackwards/hdfs-inotify-zookeeper
> >
>

Re: [DISCUSS] Move SHELL type functions from management to stellar common

[Otto Fowler]

And File loading, as casey stated


On January 31, 2018 at 10:18:22, Otto Fowler (ottobackwa...@gmail.com)
wrote:

Yeah,

ShellFunctions  to start.

Later, KafkaFunctions
Then GrokFunctions.  I would propose to split the loading of grok out from
the eval for that etc.



On January 31, 2018 at 10:14:45, Casey Stella (ceste...@gmail.com) wrote:

I assumed he was talking about the SHELL_EDIT stuff and maybe the file
loading bits. The config stuff is metron specific

On Wed, Jan 31, 2018 at 10:06 AM, Nick Allen  wrote:

> > I think we should move the shell/console type functions from stellar
>
> What functions specifically? Are you talking about
> `metron-platform/metron-management`?
>
> If you are talking about he functions like 'CONFIG_GET' and 'CONFIG_PUT',
> those seem specific to interacting with Metron. They don't seem very
> stellar-common-ish to me.
>
>
> > ... and guard them with CONSOLE capability.
>
> I think some sort of guard is appropriate. However it is done you need to
> make sure they still work in Zeppelin. There is no CONSOLE capability in
> Zeppelin, currently.
>
>
>
> On Wed, Jan 31, 2018 at 9:12 AM, Otto Fowler 
> wrote:
>
> > Per: https://issues.apache.org/jira/browse/METRON-876
> >
> > I think we should move the shell/console type functions from stellar
> > management to stellar-common, and guard them with CONSOLE capability.
> > Thoughts?
> >
> > ottO
> >
>

Re: [DISCUSS] Move SHELL type functions from management to stellar common

[Otto Fowler]

Yeah,

ShellFunctions  to start.

Later, KafkaFunctions
Then GrokFunctions.  I would propose to split the loading of grok out from
the eval for that etc.



On January 31, 2018 at 10:14:45, Casey Stella (ceste...@gmail.com) wrote:

I assumed he was talking about the SHELL_EDIT stuff and maybe the file
loading bits. The config stuff is metron specific

On Wed, Jan 31, 2018 at 10:06 AM, Nick Allen  wrote:

> > I think we should move the shell/console type functions from stellar
>
> What functions specifically? Are you talking about
> `metron-platform/metron-management`?
>
> If you are talking about he functions like 'CONFIG_GET' and 'CONFIG_PUT',
> those seem specific to interacting with Metron. They don't seem very
> stellar-common-ish to me.
>
>
> > ... and guard them with CONSOLE capability.
>
> I think some sort of guard is appropriate. However it is done you need to
> make sure they still work in Zeppelin. There is no CONSOLE capability in
> Zeppelin, currently.
>
>
>
> On Wed, Jan 31, 2018 at 9:12 AM, Otto Fowler 
> wrote:
>
> > Per: https://issues.apache.org/jira/browse/METRON-876
> >
> > I think we should move the shell/console type functions from stellar
> > management to stellar-common, and guard them with CONSOLE capability.
> > Thoughts?
> >
> > ottO
> >
>

Re: [DISCUSS] Move SHELL type functions from management to stellar common

[Casey Stella]

I assumed he was talking about the SHELL_EDIT stuff and maybe the file
loading bits.  The config stuff is metron specific

On Wed, Jan 31, 2018 at 10:06 AM, Nick Allen  wrote:

> > I think we should move the shell/console type functions from stellar
>
> What functions specifically?  Are you talking about
> `metron-platform/metron-management`?
>
> If you are talking about he functions like 'CONFIG_GET' and 'CONFIG_PUT',
> those seem specific to interacting with Metron.  They don't seem very
> stellar-common-ish to me.
>
>
> > ... and guard them with CONSOLE capability.
>
> I think some sort of guard is appropriate. However it is done you need to
> make sure they still work in Zeppelin.  There is no CONSOLE capability in
> Zeppelin, currently.
>
>
>
> On Wed, Jan 31, 2018 at 9:12 AM, Otto Fowler 
> wrote:
>
> > Per:  https://issues.apache.org/jira/browse/METRON-876
> >
> > I think we should move the shell/console type functions from stellar
> > management to stellar-common, and guard them with CONSOLE capability.
> > Thoughts?
> >
> > ottO
> >
>

Re: [DISCUSS] Move SHELL type functions from management to stellar common

[Nick Allen]

> I think we should move the shell/console type functions from stellar

What functions specifically?  Are you talking about
`metron-platform/metron-management`?

If you are talking about he functions like 'CONFIG_GET' and 'CONFIG_PUT',
those seem specific to interacting with Metron.  They don't seem very
stellar-common-ish to me.


> ... and guard them with CONSOLE capability.

I think some sort of guard is appropriate. However it is done you need to
make sure they still work in Zeppelin.  There is no CONSOLE capability in
Zeppelin, currently.



On Wed, Jan 31, 2018 at 9:12 AM, Otto Fowler 
wrote:

> Per:  https://issues.apache.org/jira/browse/METRON-876
>
> I think we should move the shell/console type functions from stellar
> management to stellar-common, and guard them with CONSOLE capability.
> Thoughts?
>
> ottO
>

[GitHub] metron issue #915: METRON-1433: Only emit debugging timing fields in enrichm...

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/915
  
Perhaps we can leverage parts of the performance logging code for this
purpose? The frequency/sampling bit was something I had wanted to add for
some time now.

https://github.com/apache/metron/pull/665


On Wed, Jan 31, 2018 at 7:58 AM, Casey Stella 
wrote:

> @mraliagha  Well, if it's being used
> explicitly and on an on-going manner, then I'll remove this PR. Do you
> think there's some value to being able to slice and dice in the global
> config the ability to turn on and off these by:
>
>- type - threatintel vs enrichment
>- adapter - stellar vs hbase vs geo
>- frequency - kinda like a tracer, have 1 in k messages have them
>
> Thoughts?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

[GitHub] metron issue #915: METRON-1433: Only emit debugging timing fields in enrichm...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/915
  
@mraliagha Well, if it's being used explicitly and on an on-going manner, 
then I'll remove this PR.  Do you think there's some value to being able to 
slice and dice in the global config the ability to turn on and off these by:
* type - threatintel vs enrichment
* adapter - stellar vs hbase vs geo
* frequency - kinda like a tracer, have 1 in k messages have them

Thoughts?


---

Re: [DISCUSS] Move SHELL type functions from management to stellar common

[Michael Miklavcic]

Agreed

On Jan 31, 2018 7:51 AM, "Justin Leet"  wrote:

> Agreed, I think it makes sense to move them there.
>
> On Wed, Jan 31, 2018 at 9:28 AM, Casey Stella  wrote:
>
> > I'd be in favor of that.  That is general purpose stuff.
> >
> > On Wed, Jan 31, 2018 at 9:12 AM, Otto Fowler 
> > wrote:
> >
> > > Per:  https://issues.apache.org/jira/browse/METRON-876
> > >
> > > I think we should move the shell/console type functions from stellar
> > > management to stellar-common, and guard them with CONSOLE capability.
> > > Thoughts?
> > >
> > > ottO
> > >
> >
>

Re: [DISCUSS] Move SHELL type functions from management to stellar common

[Justin Leet]

Agreed, I think it makes sense to move them there.

On Wed, Jan 31, 2018 at 9:28 AM, Casey Stella  wrote:

> I'd be in favor of that.  That is general purpose stuff.
>
> On Wed, Jan 31, 2018 at 9:12 AM, Otto Fowler 
> wrote:
>
> > Per:  https://issues.apache.org/jira/browse/METRON-876
> >
> > I think we should move the shell/console type functions from stellar
> > management to stellar-common, and guard them with CONSOLE capability.
> > Thoughts?
> >
> > ottO
> >
>

Re: [DISCUSS] Move SHELL type functions from management to stellar common

[Casey Stella]

I'd be in favor of that.  That is general purpose stuff.

On Wed, Jan 31, 2018 at 9:12 AM, Otto Fowler 
wrote:

> Per:  https://issues.apache.org/jira/browse/METRON-876
>
> I think we should move the shell/console type functions from stellar
> management to stellar-common, and guard them with CONSOLE capability.
> Thoughts?
>
> ottO
>

[DISCUSS] Move SHELL type functions from management to stellar common

[Otto Fowler]

Per:  https://issues.apache.org/jira/browse/METRON-876

I think we should move the shell/console type functions from stellar
management to stellar-common, and guard them with CONSOLE capability.
Thoughts?

ottO

[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/857
  
What is the status of this pr?  it is 29 day without comment, and 
conflicted, literally, and perhaps figuratively



---

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/856#discussion_r165048687
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/validation/StellarZookeeperBasedValidator.java
 ---
@@ -0,0 +1,106 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.stellar.common.utils.validation;
+
+import static 
org.apache.metron.stellar.common.shell.StellarShell.ERROR_PROMPT;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import org.apache.commons.lang.NullArgumentException;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StellarZookeeperBasedValidator implements StellarValidator {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+  private static final String FAILED_COMPILE = "Failed to compile";
+  private CuratorFramework client;
+
+  public StellarZookeeperBasedValidator(CuratorFramework client) throws 
NullArgumentException {
+if (client == null) {
+  throw new NullArgumentException("client");
+}
+this.client = client;
+  }
+
+
+  @Override
+  public Iterable validate(Optional writer) {
+// discover all the StellarConfigurationProvider
+Set providerSet = new HashSet<>();
+
+for (Class c : 
ClassIndex.getSubclasses(StellarConfigurationProvider.class,
--- End diff --

@cestella ping


---