[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr closed the pull request at:

https://github.com/apache/metron/pull/911


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165142091
  
--- Diff: 
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/integration/components/SolrComponent.java
 ---
@@ -158,4 +162,16 @@ public boolean hasCollection(String collection) {
 }
 return docs;
   }
+
+  public void addDocs(String collection, List> docs)
+  throws IOException, SolrServerException {
+CloudSolrClient solr = miniSolrCloudCluster.getSolrClient();
+solr.setDefaultCollection(collection);
+Collection solrInputDocuments = 
docs.stream().map(doc -> {
+  SolrInputDocument solrInputDocument = new SolrInputDocument();
+  doc.entrySet().forEach(entry -> 
solrInputDocument.addField(entry.getKey(), entry.getValue()));
--- End diff --

Done


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165141979
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.Group;
+import org.apache.metron.indexing.dao.search.GroupOrder;
+import org.apache.metron.indexing.dao.search.GroupOrderType;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.GroupResult;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchDao;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.search.SearchResult;
+import org.apache.metron.indexing.dao.search.SortField;
+import org.apache.metron.indexing.dao.search.SortOrder;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.client.solrj.response.FacetField.Count;
+import org.apache.solr.client.solrj.response.PivotField;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrSearchDao implements SearchDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+  private AccessConfig accessConfig;
+
+  public SolrSearchDao(SolrClient client, AccessConfig accessConfig) {
+this.client = client;
+this.accessConfig = accessConfig;
+  }
+
+  @Override
+  public SearchResponse search(SearchRequest searchRequest) throws 
InvalidSearchException {
+if (searchRequest.getQuery() == null) {
+  throw new InvalidSearchException("Search query is invalid: null");
+}
+if (client == null) {
+  throw new InvalidSearchException("Uninitialized Dao!  You must call 
init() prior to use.");
+}
+if (searchRequest.getSize() > accessConfig.getMaxSearchResults()) {
+  throw new InvalidSearchException(
+  "Search result size must be less than " + 
accessConfig.getMaxSearchResults());
+}
+SolrQuery query = buildSearchRequest(searchRequest);
+try {
+  QueryResponse response = client.query(query);
+  return buildSearchResponse(searchRequest, response);
+} catch (IOException | SolrServerException e) {
+  String msg = e.getMessage();
+  LOG.error(msg, e);
+  throw new InvalidSearchException(msg, e);
+}
+  }
+
+  @Override
+  public GroupResponse group(GroupRequest groupRequest) throws 
InvalidSearchException {
+String groupNames = 

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165142049
  
--- Diff: 
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/integration/SolrSearchIntegrationTest.java
 ---
@@ -0,0 +1,152 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.integration;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.SearchIntegrationTest;
+import org.apache.metron.indexing.dao.search.FieldType;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.integration.InMemoryComponent;
+import org.apache.metron.solr.dao.SolrDao;
+import org.apache.metron.solr.integration.components.SolrComponent;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.json.simple.JSONArray;
+import org.json.simple.parser.JSONParser;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class SolrSearchIntegrationTest extends SearchIntegrationTest {
+
+  private SolrComponent solrComponent;
+
+  @Override
+  protected IndexDao createDao() throws Exception {
+AccessConfig config = new AccessConfig();
+config.setMaxSearchResults(100);
+config.setMaxSearchGroups(100);
+config.setGlobalConfigSupplier( () ->
+new HashMap() {{
+  put("solr.zookeeper", solrComponent.getZookeeperUrl());
+}}
+);
+
+IndexDao dao = new SolrDao();
+dao.init(config);
+return dao;
+  }
+
+  @Override
+  protected InMemoryComponent startIndex() throws Exception {
+solrComponent = new SolrComponent.Builder()
+.addCollection("bro", 
"../metron-solr/src/test/resources/config/bro/conf")
+.addCollection("snort", 
"../metron-solr/src/test/resources/config/snort/conf")
+.build();
+solrComponent.start();
+return solrComponent;
+  }
+
+  @Override
--- End diff --

Done


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165141915
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.Group;
+import org.apache.metron.indexing.dao.search.GroupOrder;
+import org.apache.metron.indexing.dao.search.GroupOrderType;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.GroupResult;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchDao;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.search.SearchResult;
+import org.apache.metron.indexing.dao.search.SortField;
+import org.apache.metron.indexing.dao.search.SortOrder;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.client.solrj.response.FacetField.Count;
+import org.apache.solr.client.solrj.response.PivotField;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrSearchDao implements SearchDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+  private AccessConfig accessConfig;
+
+  public SolrSearchDao(SolrClient client, AccessConfig accessConfig) {
+this.client = client;
+this.accessConfig = accessConfig;
+  }
+
+  @Override
+  public SearchResponse search(SearchRequest searchRequest) throws 
InvalidSearchException {
+if (searchRequest.getQuery() == null) {
+  throw new InvalidSearchException("Search query is invalid: null");
+}
+if (client == null) {
+  throw new InvalidSearchException("Uninitialized Dao!  You must call 
init() prior to use.");
+}
+if (searchRequest.getSize() > accessConfig.getMaxSearchResults()) {
+  throw new InvalidSearchException(
+  "Search result size must be less than " + 
accessConfig.getMaxSearchResults());
+}
+SolrQuery query = buildSearchRequest(searchRequest);
+try {
+  QueryResponse response = client.query(query);
+  return buildSearchResponse(searchRequest, response);
+} catch (IOException | SolrServerException e) {
+  String msg = e.getMessage();
+  LOG.error(msg, e);
+  throw new InvalidSearchException(msg, e);
+}
+  }
+
+  @Override
+  public GroupResponse group(GroupRequest groupRequest) throws 
InvalidSearchException {
+String groupNames = 

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165141548
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.Group;
+import org.apache.metron.indexing.dao.search.GroupOrder;
+import org.apache.metron.indexing.dao.search.GroupOrderType;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.GroupResult;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchDao;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.search.SearchResult;
+import org.apache.metron.indexing.dao.search.SortField;
+import org.apache.metron.indexing.dao.search.SortOrder;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.client.solrj.response.FacetField.Count;
+import org.apache.solr.client.solrj.response.PivotField;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrSearchDao implements SearchDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+  private AccessConfig accessConfig;
+
+  public SolrSearchDao(SolrClient client, AccessConfig accessConfig) {
+this.client = client;
+this.accessConfig = accessConfig;
+  }
+
+  @Override
+  public SearchResponse search(SearchRequest searchRequest) throws 
InvalidSearchException {
+if (searchRequest.getQuery() == null) {
+  throw new InvalidSearchException("Search query is invalid: null");
+}
+if (client == null) {
+  throw new InvalidSearchException("Uninitialized Dao!  You must call 
init() prior to use.");
+}
+if (searchRequest.getSize() > accessConfig.getMaxSearchResults()) {
+  throw new InvalidSearchException(
+  "Search result size must be less than " + 
accessConfig.getMaxSearchResults());
+}
+SolrQuery query = buildSearchRequest(searchRequest);
+try {
+  QueryResponse response = client.query(query);
+  return buildSearchResponse(searchRequest, response);
+} catch (IOException | SolrServerException e) {
+  String msg = e.getMessage();
+  LOG.error(msg, e);
+  throw new InvalidSearchException(msg, e);
+}
+  }
+
+  @Override
+  public GroupResponse group(GroupRequest groupRequest) throws 
InvalidSearchException {
+String groupNames = 

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165141296
  
--- Diff: 
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java
 ---
@@ -655,83 +699,54 @@ public void disabled_facet_query_returns_null_count() 
throws Exception {
   }
 
   @Test
-  public void exceeding_max_resulsts_throws_exception() throws Exception {
+  public void missing_type_facet_query() throws Exception {
+SearchRequest request = JSONUtils.INSTANCE.load(missingTypeFacetQuery, 
SearchRequest.class);
+SearchResponse response = dao.search(request);
+Assert.assertEquals(10, response.getTotal());
+
+Map> facetCounts = response.getFacetCounts();
+Assert.assertEquals(1, facetCounts.size());
+Map snortFieldCounts = facetCounts.get("snort_field");
+Assert.assertEquals(5, snortFieldCounts.size());
+Assert.assertEquals(new Long(1), snortFieldCounts.get("50"));
--- End diff --

Done


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r165141322
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrDao.java
 ---
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.ColumnMetadataDao;
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.search.FieldType;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrDao implements IndexDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+  public static final String ID_FIELD = "guid";
--- End diff --

Done


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164881484
  
--- Diff: 
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java
 ---
@@ -443,11 +495,11 @@ public void all_query_returns_all_results() throws 
Exception {
 Assert.assertEquals(10, results.size());
 for(int i = 0;i < 5;++i) {
   Assert.assertEquals("snort", 
results.get(i).getSource().get("source:type"));
-  Assert.assertEquals(10 - i, 
results.get(i).getSource().get("timestamp"));
+  Assert.assertEquals(10 - i + "", 
results.get(i).getSource().get("timestamp").toString());
--- End diff --

The ES and Solr clients return different types (Int vs Longs) so I am 
converting to string to account for that.  I'm not sure this is the correct 
approach and open to ideas.


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164881237
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/search/SearchDao.java
 ---
@@ -0,0 +1,34 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.indexing.dao.search;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.metron.indexing.dao.update.Document;
+
+public interface SearchDao {
--- End diff --

I was undecided as to whether we should do it in this PR or a future one 
and also wanted to make sure people liked it.  Will make this change.


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164847601
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.Group;
+import org.apache.metron.indexing.dao.search.GroupOrder;
+import org.apache.metron.indexing.dao.search.GroupOrderType;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.GroupResult;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchDao;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.search.SearchResult;
+import org.apache.metron.indexing.dao.search.SortField;
+import org.apache.metron.indexing.dao.search.SortOrder;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.client.solrj.response.FacetField.Count;
+import org.apache.solr.client.solrj.response.PivotField;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrSearchDao implements SearchDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+  private AccessConfig accessConfig;
+
+  public SolrSearchDao(SolrClient client, AccessConfig accessConfig) {
+this.client = client;
+this.accessConfig = accessConfig;
+  }
+
+  @Override
+  public SearchResponse search(SearchRequest searchRequest) throws 
InvalidSearchException {
+if (searchRequest.getQuery() == null) {
+  throw new InvalidSearchException("Search query is invalid: null");
+}
+if (client == null) {
+  throw new InvalidSearchException("Uninitialized Dao!  You must call 
init() prior to use.");
+}
+if (searchRequest.getSize() > accessConfig.getMaxSearchResults()) {
+  throw new InvalidSearchException(
+  "Search result size must be less than " + 
accessConfig.getMaxSearchResults());
+}
+SolrQuery query = buildSearchRequest(searchRequest);
+try {
+  QueryResponse response = client.query(query);
+  return buildSearchResponse(searchRequest, response);
+} catch (IOException | SolrServerException e) {
+  String msg = e.getMessage();
+  LOG.error(msg, e);
+  throw new InvalidSearchException(msg, e);
+}
+  }
+
+  @Override
+  public GroupResponse group(GroupRequest groupRequest) throws 
InvalidSearchException {
+String groupNames = 

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164842112
  
--- Diff: 
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/integration/SolrSearchIntegrationTest.java
 ---
@@ -0,0 +1,152 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.integration;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.SearchIntegrationTest;
+import org.apache.metron.indexing.dao.search.FieldType;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.integration.InMemoryComponent;
+import org.apache.metron.solr.dao.SolrDao;
+import org.apache.metron.solr.integration.components.SolrComponent;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.json.simple.JSONArray;
+import org.json.simple.parser.JSONParser;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class SolrSearchIntegrationTest extends SearchIntegrationTest {
+
+  private SolrComponent solrComponent;
+
+  @Override
+  protected IndexDao createDao() throws Exception {
+AccessConfig config = new AccessConfig();
+config.setMaxSearchResults(100);
+config.setMaxSearchGroups(100);
+config.setGlobalConfigSupplier( () ->
+new HashMap() {{
+  put("solr.zookeeper", solrComponent.getZookeeperUrl());
+}}
+);
+
+IndexDao dao = new SolrDao();
+dao.init(config);
+return dao;
+  }
+
+  @Override
+  protected InMemoryComponent startIndex() throws Exception {
+solrComponent = new SolrComponent.Builder()
+.addCollection("bro", 
"../metron-solr/src/test/resources/config/bro/conf")
+.addCollection("snort", 
"../metron-solr/src/test/resources/config/snort/conf")
+.build();
+solrComponent.start();
+return solrComponent;
+  }
+
+  @Override
--- End diff --

Could you throw
```
@SuppressWarnings("unchecked")
```
on here?


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164840018
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.Group;
+import org.apache.metron.indexing.dao.search.GroupOrder;
+import org.apache.metron.indexing.dao.search.GroupOrderType;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.GroupResult;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchDao;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.search.SearchResult;
+import org.apache.metron.indexing.dao.search.SortField;
+import org.apache.metron.indexing.dao.search.SortOrder;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.client.solrj.response.FacetField.Count;
+import org.apache.solr.client.solrj.response.PivotField;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrSearchDao implements SearchDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+  private AccessConfig accessConfig;
+
+  public SolrSearchDao(SolrClient client, AccessConfig accessConfig) {
+this.client = client;
+this.accessConfig = accessConfig;
+  }
+
+  @Override
+  public SearchResponse search(SearchRequest searchRequest) throws 
InvalidSearchException {
+if (searchRequest.getQuery() == null) {
+  throw new InvalidSearchException("Search query is invalid: null");
+}
+if (client == null) {
+  throw new InvalidSearchException("Uninitialized Dao!  You must call 
init() prior to use.");
+}
+if (searchRequest.getSize() > accessConfig.getMaxSearchResults()) {
+  throw new InvalidSearchException(
+  "Search result size must be less than " + 
accessConfig.getMaxSearchResults());
+}
+SolrQuery query = buildSearchRequest(searchRequest);
+try {
+  QueryResponse response = client.query(query);
+  return buildSearchResponse(searchRequest, response);
+} catch (IOException | SolrServerException e) {
+  String msg = e.getMessage();
+  LOG.error(msg, e);
+  throw new InvalidSearchException(msg, e);
+}
+  }
+
+  @Override
+  public GroupResponse group(GroupRequest groupRequest) throws 
InvalidSearchException {
+String groupNames = 

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164839532
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.Group;
+import org.apache.metron.indexing.dao.search.GroupOrder;
+import org.apache.metron.indexing.dao.search.GroupOrderType;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.GroupResult;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchDao;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.search.SearchResult;
+import org.apache.metron.indexing.dao.search.SortField;
+import org.apache.metron.indexing.dao.search.SortOrder;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.client.solrj.response.FacetField.Count;
+import org.apache.solr.client.solrj.response.PivotField;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrSearchDao implements SearchDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+  private AccessConfig accessConfig;
+
+  public SolrSearchDao(SolrClient client, AccessConfig accessConfig) {
+this.client = client;
+this.accessConfig = accessConfig;
+  }
+
+  @Override
+  public SearchResponse search(SearchRequest searchRequest) throws 
InvalidSearchException {
+if (searchRequest.getQuery() == null) {
+  throw new InvalidSearchException("Search query is invalid: null");
+}
+if (client == null) {
+  throw new InvalidSearchException("Uninitialized Dao!  You must call 
init() prior to use.");
+}
+if (searchRequest.getSize() > accessConfig.getMaxSearchResults()) {
+  throw new InvalidSearchException(
+  "Search result size must be less than " + 
accessConfig.getMaxSearchResults());
+}
+SolrQuery query = buildSearchRequest(searchRequest);
+try {
+  QueryResponse response = client.query(query);
+  return buildSearchResponse(searchRequest, response);
+} catch (IOException | SolrServerException e) {
+  String msg = e.getMessage();
+  LOG.error(msg, e);
+  throw new InvalidSearchException(msg, e);
+}
+  }
+
+  @Override
+  public GroupResponse group(GroupRequest groupRequest) throws 
InvalidSearchException {
+String groupNames = 

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164841557
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrUpdateDao.java
 ---
@@ -0,0 +1,101 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Optional;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.metron.indexing.dao.update.UpdateDao;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.common.SolrInputDocument;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrUpdateDao implements UpdateDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+
+  public SolrUpdateDao(SolrClient client) {
+this.client = client;
+  }
+
+  @Override
+  public void update(Document update, Optional index) throws 
IOException {
+try {
+  SolrInputDocument solrInputDocument = toSolrInputDocument(update);
+  if (index.isPresent()) {
+this.client.add(index.get(), solrInputDocument);
+  } else {
+this.client.add(solrInputDocument);
+  }
+} catch (SolrServerException e) {
+  throw new IOException(e);
+}
+  }
+
+  @Override
+  public void batchUpdate(Map updates) throws 
IOException {
+// updates with a collection specified
+Map solrCollectionUpdates = new 
HashMap<>();
+
+// updates with no collection specified
+Collection solrUpdates = new ArrayList<>();
+
+for(Entry entry: updates.entrySet()) {
+  SolrInputDocument solrInputDocument = 
toSolrInputDocument(entry.getKey());
+  Optional index = entry.getValue();
+  if (index.isPresent()) {
+Collection solrInputDocuments = 
solrCollectionUpdates.get(index.get());
+if (solrInputDocuments == null) {
+  solrInputDocuments = new ArrayList<>();
+}
+solrInputDocuments.add(solrInputDocument);
+solrCollectionUpdates.put(index.get(), solrInputDocuments);
+  } else {
+solrUpdates.add(solrInputDocument);
+  }
+}
+try {
+  if (!solrCollectionUpdates.isEmpty()) {
+for(Entry entry: 
solrCollectionUpdates.entrySet()) {
+  this.client.add(entry.getKey(), entry.getValue());
+}
+  } else {
+this.client.add(solrUpdates);
+  }
+} catch (SolrServerException e) {
+  throw new IOException(e);
+}
+  }
+
+  private SolrInputDocument toSolrInputDocument(Document document) {
+SolrInputDocument solrInputDocument = new SolrInputDocument();
+document.getDocument().entrySet().forEach(entry ->
--- End diff --

Can just be
```
 document.getDocument().forEach(solrInputDocument::addField);
```


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164844818
  
--- Diff: 
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/search/SearchDao.java
 ---
@@ -0,0 +1,34 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.indexing.dao.search;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.metron.indexing.dao.update.Document;
+
+public interface SearchDao {
--- End diff --

Seems like if we're adding this (and I do like having this split out by 
functionality), it should also be applied to the Elasticsearch Daos (same for 
things like UpdateDao).

I'd also strongly be in favor of splitting out the ElasticsearchDao as part 
of this, to make sure that the overall structure of the two Dao sets are very 
comparable and easy to understand.  The integration tests should be able to 
insulate us from the obvious issues, and IMO it's worthwhile to do now.


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164840108
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.Group;
+import org.apache.metron.indexing.dao.search.GroupOrder;
+import org.apache.metron.indexing.dao.search.GroupOrderType;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.GroupResult;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchDao;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.search.SearchResult;
+import org.apache.metron.indexing.dao.search.SortField;
+import org.apache.metron.indexing.dao.search.SortOrder;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.client.solrj.response.FacetField.Count;
+import org.apache.solr.client.solrj.response.PivotField;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrSearchDao implements SearchDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+  private AccessConfig accessConfig;
+
+  public SolrSearchDao(SolrClient client, AccessConfig accessConfig) {
+this.client = client;
+this.accessConfig = accessConfig;
+  }
+
+  @Override
+  public SearchResponse search(SearchRequest searchRequest) throws 
InvalidSearchException {
+if (searchRequest.getQuery() == null) {
+  throw new InvalidSearchException("Search query is invalid: null");
+}
+if (client == null) {
+  throw new InvalidSearchException("Uninitialized Dao!  You must call 
init() prior to use.");
+}
+if (searchRequest.getSize() > accessConfig.getMaxSearchResults()) {
+  throw new InvalidSearchException(
+  "Search result size must be less than " + 
accessConfig.getMaxSearchResults());
+}
+SolrQuery query = buildSearchRequest(searchRequest);
+try {
+  QueryResponse response = client.query(query);
+  return buildSearchResponse(searchRequest, response);
+} catch (IOException | SolrServerException e) {
+  String msg = e.getMessage();
+  LOG.error(msg, e);
+  throw new InvalidSearchException(msg, e);
+}
+  }
+
+  @Override
+  public GroupResponse group(GroupRequest groupRequest) throws 
InvalidSearchException {
+String groupNames = 

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164841286
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.Group;
+import org.apache.metron.indexing.dao.search.GroupOrder;
+import org.apache.metron.indexing.dao.search.GroupOrderType;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.GroupResult;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchDao;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.search.SearchResult;
+import org.apache.metron.indexing.dao.search.SortField;
+import org.apache.metron.indexing.dao.search.SortOrder;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.client.solrj.response.FacetField.Count;
+import org.apache.solr.client.solrj.response.PivotField;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrSearchDao implements SearchDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+  private AccessConfig accessConfig;
+
+  public SolrSearchDao(SolrClient client, AccessConfig accessConfig) {
+this.client = client;
+this.accessConfig = accessConfig;
+  }
+
+  @Override
+  public SearchResponse search(SearchRequest searchRequest) throws 
InvalidSearchException {
+if (searchRequest.getQuery() == null) {
+  throw new InvalidSearchException("Search query is invalid: null");
+}
+if (client == null) {
+  throw new InvalidSearchException("Uninitialized Dao!  You must call 
init() prior to use.");
+}
+if (searchRequest.getSize() > accessConfig.getMaxSearchResults()) {
+  throw new InvalidSearchException(
+  "Search result size must be less than " + 
accessConfig.getMaxSearchResults());
+}
+SolrQuery query = buildSearchRequest(searchRequest);
+try {
+  QueryResponse response = client.query(query);
+  return buildSearchResponse(searchRequest, response);
+} catch (IOException | SolrServerException e) {
+  String msg = e.getMessage();
+  LOG.error(msg, e);
+  throw new InvalidSearchException(msg, e);
+}
+  }
+
+  @Override
+  public GroupResponse group(GroupRequest groupRequest) throws 
InvalidSearchException {
+String groupNames = 

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164842946
  
--- Diff: 
metron-platform/metron-solr/src/test/java/org/apache/metron/solr/integration/components/SolrComponent.java
 ---
@@ -158,4 +162,16 @@ public boolean hasCollection(String collection) {
 }
 return docs;
   }
+
+  public void addDocs(String collection, List> docs)
+  throws IOException, SolrServerException {
+CloudSolrClient solr = miniSolrCloudCluster.getSolrClient();
+solr.setDefaultCollection(collection);
+Collection solrInputDocuments = 
docs.stream().map(doc -> {
+  SolrInputDocument solrInputDocument = new SolrInputDocument();
+  doc.entrySet().forEach(entry -> 
solrInputDocument.addField(entry.getKey(), entry.getValue()));
--- End diff --

This line can be
```
  doc.forEach(solrInputDocument::addField);
```


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164846538
  
--- Diff: 
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java
 ---
@@ -443,11 +495,11 @@ public void all_query_returns_all_results() throws 
Exception {
 Assert.assertEquals(10, results.size());
 for(int i = 0;i < 5;++i) {
   Assert.assertEquals("snort", 
results.get(i).getSource().get("source:type"));
-  Assert.assertEquals(10 - i, 
results.get(i).getSource().get("timestamp"));
+  Assert.assertEquals(10 - i + "", 
results.get(i).getSource().get("timestamp").toString());
--- End diff --

Why are all the timestamps strings now?


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164838120
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrDao.java
 ---
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.ColumnMetadataDao;
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.search.FieldType;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrDao implements IndexDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+  public static final String ID_FIELD = "guid";
--- End diff --

We already have GUID defined in Constants, we should use that.


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164838944
  
--- Diff: 
metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
 ---
@@ -0,0 +1,315 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.solr.dao;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.stream.Collectors;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.search.GetRequest;
+import org.apache.metron.indexing.dao.search.Group;
+import org.apache.metron.indexing.dao.search.GroupOrder;
+import org.apache.metron.indexing.dao.search.GroupOrderType;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.GroupResult;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchDao;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.search.SearchResult;
+import org.apache.metron.indexing.dao.search.SortField;
+import org.apache.metron.indexing.dao.search.SortOrder;
+import org.apache.metron.indexing.dao.update.Document;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.SolrQuery.ORDER;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.response.FacetField;
+import org.apache.solr.client.solrj.response.FacetField.Count;
+import org.apache.solr.client.solrj.response.PivotField;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SolrSearchDao implements SearchDao {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  private transient SolrClient client;
+  private AccessConfig accessConfig;
+
+  public SolrSearchDao(SolrClient client, AccessConfig accessConfig) {
+this.client = client;
+this.accessConfig = accessConfig;
+  }
+
+  @Override
+  public SearchResponse search(SearchRequest searchRequest) throws 
InvalidSearchException {
+if (searchRequest.getQuery() == null) {
+  throw new InvalidSearchException("Search query is invalid: null");
+}
+if (client == null) {
+  throw new InvalidSearchException("Uninitialized Dao!  You must call 
init() prior to use.");
+}
+if (searchRequest.getSize() > accessConfig.getMaxSearchResults()) {
+  throw new InvalidSearchException(
+  "Search result size must be less than " + 
accessConfig.getMaxSearchResults());
+}
+SolrQuery query = buildSearchRequest(searchRequest);
+try {
+  QueryResponse response = client.query(query);
+  return buildSearchResponse(searchRequest, response);
+} catch (IOException | SolrServerException e) {
+  String msg = e.getMessage();
+  LOG.error(msg, e);
+  throw new InvalidSearchException(msg, e);
+}
+  }
+
+  @Override
+  public GroupResponse group(GroupRequest groupRequest) throws 
InvalidSearchException {
+String groupNames = 

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/911#discussion_r164836556
  
--- Diff: 
metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/SearchIntegrationTest.java
 ---
@@ -655,83 +699,54 @@ public void disabled_facet_query_returns_null_count() 
throws Exception {
   }
 
   @Test
-  public void exceeding_max_resulsts_throws_exception() throws Exception {
+  public void missing_type_facet_query() throws Exception {
+SearchRequest request = JSONUtils.INSTANCE.load(missingTypeFacetQuery, 
SearchRequest.class);
+SearchResponse response = dao.search(request);
+Assert.assertEquals(10, response.getTotal());
+
+Map> facetCounts = response.getFacetCounts();
+Assert.assertEquals(1, facetCounts.size());
+Map snortFieldCounts = facetCounts.get("snort_field");
+Assert.assertEquals(5, snortFieldCounts.size());
+Assert.assertEquals(new Long(1), snortFieldCounts.get("50"));
--- End diff --

any reason these aren't just 1L?


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

Github user merrimanr closed the pull request at:

https://github.com/apache/metron/pull/911


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

GitHub user merrimanr reopened a pull request:

https://github.com/apache/metron/pull/911

METRON-1419: Create a SolrDao

## Contributor Comments
This PR is an initial attempt at creating a SolrDao that implements the 
IndexDao interface, is functionally equivalent to ElasticsearchDao and passes 
all tests in SearchIntegrationTest and UpdateIntegrationTest.  

A high level summary of the changes include:

- Upgraded the Solr client version to 6.6.0
- Updated the SolrComponent to work with Solr 6.6 and added a couple 
convenience methods similar to ElasticsearchComponent
- Added a new SolrDao implementation with all IndexDao methods implemented
- Refactored the SearchIntegrationTest to work for both Solr and 
Elasticsearch and added an Solr implementation (more detail below)
- Created an abstract UpdateIntegrationTest and added a Solr implementation
- Added Solr schemas for test data sets
- Added new tests to SearchIntegrationTest including filtering on fields 
with different types, faceting on fields with different types, and faceting on 
fields with missing types.
- Broke the IndexDao down in the SolrDao to smaller, easier to understand 
classes.  The ElasticsearchDao class has become very large so I attempted to 
make the SolrDao more readable.

There were a couple areas where Elasticsearch and Solr behave slightly 
different.  I attempted to accommodated for that through the SolrDao 
implementation, by adjusting existing tests, and by splitting out specific 
tests:

- Column metadata is different between the 2 search engines so each 
implementation has their own tests
- There are cases where the clients will return different types in search 
results.  I am handling this in SearchIntegrationTest by first converting the 
types to strings and then comparing (other ideas?).  For example, the ES client 
returns an Integer for timestamp while Solr returns a Long.
- There are cases where ES throws an error under certain conditions while 
Solr does not (and vice-versa).  These were moved to either ES or Solr 
SearchIntegrationTest implementations.
- There is no support in Solr for sorting group results so I am sorting 
them client-side instead.

At this point the scope is limited to tests passing, meaning 
metron-elasticsearch and metron-solr pass all tests.  There are other PRs in 
progress that are needed before automated testing with full dev can be done.  I 
am still actively working on manually testing in full dev and adding 
documentation but this should get us started.

This PR is intended to be merged into the upgrade-solr feature branch but I 
have it set to master temporarily so review is easier.  We will need to merge 
in master to the feature branch to get rid of the extra commits since this PR 
is up to date with master.

I'm expecting a lengthy review and would request multiple reviewers and +1s.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant 

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

[merrimanr]

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/911

METRON-1419: Create a SolrDao

## Contributor Comments
This PR is an initial attempt at creating a SolrDao that implements the 
IndexDao interface, is functionally equivalent to ElasticsearchDao and passes 
all tests in SearchIntegrationTest and UpdateIntegrationTest.  

A high level summary of the changes include:

- Upgraded the Solr client version to 6.6.0
- Updated the SolrComponent to work with Solr 6.6 and added a couple 
convenience methods similar to ElasticsearchComponent
- Added a new SolrDao implementation with all IndexDao methods implemented
- Refactored the SearchIntegrationTest to work for both Solr and 
Elasticsearch and added an Solr implementation (more detail below)
- Created an abstract UpdateIntegrationTest and added a Solr implementation
- Added Solr schemas for test data sets
- Added new tests to SearchIntegrationTest including filtering on fields 
with different types, faceting on fields with different types, and faceting on 
fields with missing types.
- Broke the IndexDao down in the SolrDao to smaller, easier to understand 
classes.  The ElasticsearchDao class has become very large so I attempted to 
make the SolrDao more readable.

There were a couple areas where Elasticsearch and Solr behave slightly 
different.  I attempted to accommodated for that through the SolrDao 
implementation, by adjusting existing tests, and by splitting out specific 
tests:

- Column metadata is different between the 2 search engines so each 
implementation has their own tests
- There are cases where the clients will return different types in search 
results.  I am handling this in SearchIntegrationTest by first converting the 
types to strings and then comparing (other ideas?).  For example, the ES client 
returns an Integer for timestamp while Solr returns a Long.
- There are cases where ES throws an error under certain conditions while 
Solr does not (and vice-versa).  These were moved to either ES or Solr 
SearchIntegrationTest implementations.
- There is no support in Solr for sorting group results so I am sorting 
them client-side instead.

At this point the scope is limited to tests passing.  There are other PRs 
in progress that are needed before automated testing with full dev can be done. 
 I am still actively working on manually testing in full dev and adding 
documentation but this should get us started.

This PR is intended to be merged into the upgrade-solr feature branch but I 
have it set to master temporarily so review is easier.  We will need to merge 
in master to the feature branch to get rid of the extra commits since this PR 
is up to date with master.

I'm expecting a lengthy review and would request multiple reviewers and +1s.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For