[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[jjmeyer0]

Github user jjmeyer0 commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126589803
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+  @Stellar(
+name = "HASH",
+description = "Hashes a given value using the given hashing algorithm 
and returns a hex encoded string. This function only hashes " +
+  "strings and values that implement java.io.Serializable.",
+params = {
+  "toHash - value to hash.",
+  "hashType - A valid string representation of an algorithm supported 
by java.security.MessageDigest.",
+},
+returns = "A hex representation of a hashed value using the given 
hashing algorithm. If either argument is " +
+  "null then null will be returned. If the type of 'toHash' is neither 
a string nor of type java.io.Serializable, " +
+  "then null is returned."
+  )
+  public static class Hash extends BaseStellarFunction {
--- End diff --

@ottobackwards I'll make sure to stay consistent with what you have.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

[mraliagha]

Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/620
  
@iraghumitra I am going to test your latest code to make sure the mentioned 
issues haven't been resolved yet. I was using an older version of your build, 
so it might be outdated.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126579811
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+  @Stellar(
+name = "HASH",
+description = "Hashes a given value using the given hashing algorithm 
and returns a hex encoded string. This function only hashes " +
+  "strings and values that implement java.io.Serializable.",
+params = {
+  "toHash - value to hash.",
+  "hashType - A valid string representation of an algorithm supported 
by java.security.MessageDigest.",
+},
+returns = "A hex representation of a hashed value using the given 
hashing algorithm. If either argument is " +
+  "null then null will be returned. If the type of 'toHash' is neither 
a string nor of type java.io.Serializable, " +
+  "then null is returned."
+  )
+  public static class Hash extends BaseStellarFunction {
--- End diff --

I changed mine to GET_ instead of LIST_ and return a List
So we shall be consistent.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126577935
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+  @Stellar(
+name = "HASH",
+description = "Hashes a given value using the given hashing algorithm 
and returns a hex encoded string. This function only hashes " +
+  "strings and values that implement java.io.Serializable.",
+params = {
+  "toHash - value to hash.",
+  "hashType - A valid string representation of an algorithm supported 
by java.security.MessageDigest.",
+},
+returns = "A hex representation of a hashed value using the given 
hashing algorithm. If either argument is " +
+  "null then null will be returned. If the type of 'toHash' is neither 
a string nor of type java.io.Serializable, " +
+  "then null is returned."
+  )
+  public static class Hash extends BaseStellarFunction {
--- End diff --

I have no preference.  I picked the string thinking of the return of the 
call being for 'information' and display only.  Like from the shell.  I think 
that we should do the 'get' no params calls the same though, i'd change to list 
if that is what makes sense


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [Request for Consensus Approval] dev branch for Stellar additional work

[Matt Foley]

Hey all, it appears that we’re not ready to do speculative branches yet, so 
I’ll proceed with the next chunk of Stellar separation work in my private 
branch, currently in https://github.com/mattf-horton/metron/tree/stellar-mod4

There is of course nothing “private” about it; anyone is most welcome to watch, 
make suggestions, and contribute PRs.

Thanks,
--Matt


On 7/5/17, 10:36 AM, "Otto Fowler"  wrote:

Yeah, this is part of why we need the guide


On July 5, 2017 at 09:23:03, zeo...@gmail.com (zeo...@gmail.com) wrote:

That all sounds pretty reasonable to me. My biggest concern would be
attribution during step 5 - we would need to make sure it isn't squash
merged like we typically do (assuming we do properly squash merge into
the speculative
branch). Not a big issue though, I guess, just need to make sure it isn't
overlooked.

Jon

On Wed, Jul 5, 2017 at 4:40 AM Matt Foley  wrote:

> Now that METRON-877 is in, I would like to proceed with Steps 3-6 of the
> remaining work to separate out Stellar functionality as an independent
> module. A couple people have suggested that this further development
> should be done in a Metron “dev branch”, where:
> a) changes are more visible than in a single person’s private development
> branch, and
> b) work can proceed for several days or a couple weeks on a branch that
> the collaborators may choose to keep stable for the duration (ie, without
> constantly updating to master).
>
> This concept was discussed as a “speculative branch” in this email
thread:
>

https://lists.apache.org/thread.html/391e15347ad625c4aa61e81f5dd238c0acb4048b8d77f93313298263@%3Cdev.metron.apache.org%3E
> but I don’t see that we ever actually changed our bylaws to mention it.
>
> Nevertheless, it falls within the purview of the PMC to create new
> branches in our code tree, and I request PMC members to give me a lazy
> consensus vote to do so. Please +1 this email if you agree.
>
> The proposed rules of engagement are (drawn from issues raised in that
> email thread):
> 1. Commits to this branch to have the same rules as to master: Jira, PR,
> and at least one +1 from a knowledgeable reviewer, and no -1’s.
> 2. +1 reviews may come from any participating contributor, not only
> current committers. But commits still have to be made by a committer, so
> we don’t have to create new auth infra for this branch.
> 3. The branch should be updated from master at least every second week,
or
> more frequently. This may be adjusted to avoid disruption of work in
> progress.
> 4. PR’s to master will be posted for review as soon as self-consistent
> chunks of useful functionality are done. The collaborators will define
> those chunks, but a rough goal is every two weeks. The goal is to avoid
> mega-patches to review.
> 5. PR’s to master will be posted by a single developer from their home
> github repo, not directly from the speculative branch, so that
> collaborative work can proceed on the speculative branch.
> 6. The PR’s will be credited equally to all collaborators active during
> that “chunk” of work.
> 7. PR’s to master have to be reviewed and agreed to as though they were
> new patches. (The fact they were previously accepted into the speculative
> branch is at most a recommendation, not an a priori decision to let them
> into master.) The usual rules apply. While collaborators will likely want
> to +1 such PRs, sufficient time must be provided for other community
> members to review and raise issues.
>
> Thanks,
> --Matt
>
>
>
> --

Jon

Re: [VOTE][PROPOSAL] minor changes to release process

[Matt Foley]

Vote passes with 
+1 : 4 votes (3 binding, 1 non-binding)
0 : none
-1 : none

I’ll edit the doc to reflect the change.
Thanks,
--Matt

On 7/6/17, 10:53 AM, "Matt Foley"  wrote:

Thanks, all.  That’s 3 binding +1’s, so I’m going to proceed with 
METRON-1021.
Vote needs to stay open 72 hours tho, so if anyone else wishes to vote pro 
or con, you’ll be listened to.
Thanks,
--Matt


On 7/6/17, 10:24 AM, "Nick Allen"  wrote:

+1  I think that makes a lot of sense.

On Wed, Jul 5, 2017 at 5:47 PM, Matt Foley  wrote:

> (The below proposal is also stated in https://issues.apache.org/
> jira/browse/METRON-1020 )
>
> The following proposed changes are small, but not just editorial in
> nature, hence will require vote of the community to change. Our bylaws
> don’t have an action type of Modifying Policy, but it’s probably fair 
to
> consider policies to be “included by reference” in Bylaws, so let’s 
vote on
> this like a Bylaws change.  “Lazy majority of PMC members” applies – 
same
> as a release.
>
> Regarding the process at https://cwiki.apache.org/
> confluence/display/METRON/Release+Process :
>
> 1. Add a step to tag the final release, as 
"apache-metron--
> release".
>
> 2. The current policy says that when a critical release is urgently
> needed, "the 72 hour waiting periods in Steps 7 and 8 can be waived." 
The
> formerly referenced Step 8 was for the Incubator vote, so that can be
> removed as an editorial issue, but we should also allow for not 
waiting for
> mirror propagation – let the mirrors catch up as fast as they can. So 
the
> text should now read: "the 72 hour waiting period in Step 7 and the 
wait
> for mirror propagation in Step 10 can be waived."
>
> 3. Finally, it is good practice to increment the build version in POMs
> immediately AFTER a release, so that builds with new stuff cannot be
> mistaken for builds of the release version. The current policy says to
> increment it just BEFORE a release. I suggest changing this to say:
> a) immediately after a release, increment the MINOR version number 
(eg,
> with the 0.4.0 just released, set the new version number to 0.4.1)
> b) immediately before a release, decide whether it will be a minor or
> major release. If minor, assure that the minor version number was 
already
> incremented after the last release and continue to use that number. If
> major, change the version number to the desired new major version.
> c) These version number changes are in master branch.  Creation of new
> branches does not occur until the idea of creating a maintenance 
branch or
> a new release branch has been consented by the community.
>
> Please share your thoughts and/or vote.
> Thanks,
> --Matt
>
>
>

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126573313
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+  @Stellar(
+name = "HASH",
+description = "Hashes a given value using the given hashing algorithm 
and returns a hex encoded string. This function only hashes " +
+  "strings and values that implement java.io.Serializable.",
+params = {
+  "toHash - value to hash.",
+  "hashType - A valid string representation of an algorithm supported 
by java.security.MessageDigest.",
+},
+returns = "A hex representation of a hashed value using the given 
hashing algorithm. If either argument is " +
+  "null then null will be returned. If the type of 'toHash' is neither 
a string nor of type java.io.Serializable, " +
+  "then null is returned."
+  )
+  public static class Hash extends BaseStellarFunction {
--- End diff --

Well, SPLIT and JOIN let you move back and forth between the two formats, 
but wouldn't the List format be the most naturally usable one?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126566101
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+  @Stellar(
+name = "HASH",
+description = "Hashes a given value using the given hashing algorithm 
and returns a hex encoded string. This function only hashes " +
+  "strings and values that implement java.io.Serializable.",
+params = {
+  "toHash - value to hash.",
+  "hashType - A valid string representation of an algorithm supported 
by java.security.MessageDigest.",
+},
+returns = "A hex representation of a hashed value using the given 
hashing algorithm. If either argument is " +
+  "null then null will be returned. If the type of 'toHash' is neither 
a string nor of type java.io.Serializable, " +
+  "then null is returned."
+  )
+  public static class Hash extends BaseStellarFunction {
--- End diff --

My pr for encoding has this, but it returns a comma delimited string


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126560557
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java
 ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+  final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+  @Test(expected = IllegalArgumentException.class)
+  public void nullArgumentListShouldThrowException() throws Exception {
+hash.apply(null);
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void emptyArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.emptyList());
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void singleArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.singletonList("some value."));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException3() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException4() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3", "4"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void invalidAlgorithmArgumentShouldThrowException() throws 
Exception {
+hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+  }
+
+  @Test
+  public void invalidNullAlgorithmArgumentShouldThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList("value to hash", null)));
+  }
+
+  @Test
+  public void nullInputForValueToHashShouldProperlyThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList(null, "md5")));
+  }
+
+  @Test
+  public void allAlgorithmsForMessageDigestShouldBeAbleToHash() throws 
Exception {
+final String valueToHash = "My value to hash";
+final Set algorithms = Security.getAlgorithms("MessageDigest");
+
+algorithms.forEach(algorithm -> {
+  try {
+final MessageDigest m = MessageDigest.getInstance(algorithm);
+m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+assertEquals(BaseEncoding.base16().encode(m.digest()), 
hash.apply(Arrays.asList(valueToHash, algorithm)));
+  } catch (NoSuchAlgorithmException e) {
+throw new RuntimeException(e);
--- End diff --

@jjmeyer0 , quite right!  Thanks, I learned something :-)
For those not already familiar with this, it isn't that it's a lambda, it's 
that it's a lambda being used as the Consumer argument to a .forEach call, and 
Consumer.accept() is contractually specified to not throw checked exceptions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/620
  
I just added some comments related to abstracting the search service.  Much 
improved with the latest commits.  I think we're almost there.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r126534924
  
--- Diff: 
metron-interface/metron-alerts/src/app/utils/elasticsearch-utils.ts ---
@@ -0,0 +1,72 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {ColumnMetadata} from '../model/column-metadata';
+import {AlertsSearchResponse} from '../model/alerts-search-response';
+
+export class ElasticsearchUtils {
+
+  private static createColumMetaData(properties: any, columnMetadata: 
ColumnMetadata[], seen: string[]) {
+ try {
+   let columnNames = Object.keys(properties);
+   for (let columnName of columnNames) {
+ if (seen.indexOf(columnName) === -1) {
+   seen.push(columnName);
+   columnMetadata.push(
+ new ColumnMetadata(columnName, (properties[columnName].type ? 
properties[columnName].type.toUpperCase() : ''))
+   );
+ }
+   }
+ } catch (e) {}
+  }
+
+  public static extractColumnNameData(res: Response): ColumnMetadata[] {
+let response: any = res || {};
+let columnMetadata: ColumnMetadata[] = [];
+let seen: string[] = [];
+
+for (let index in response.metadata.indices) {
+  if (index.startsWith('bro') || index.startsWith('bro') || 
index.startsWith('bro')) {
--- End diff --

Is this a mistake?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r126534489
  
--- Diff: 
metron-interface/metron-alerts/src/app/service/elasticsearch-localstorage-impl.ts
 ---
@@ -0,0 +1,291 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Observable} from 'rxjs/Rx';
+import {Headers, RequestOptions} from '@angular/http';
+
+import {HttpUtil} from '../utils/httpUtil';
+import {DataSource} from './data-source';
+import {Alert} from '../model/alert';
+import {ColumnMetadata} from '../model/column-metadata';
+import {ElasticsearchUtils} from '../utils/elasticsearch-utils';
+import {
+  ALERTS_COLUMN_NAMES, ALERTS_TABLE_METADATA, ALERTS_RECENT_SEARCH,
+  ALERTS_SAVED_SEARCH, NUM_SAVED_SEARCH
+} from '../utils/constants';
+import {ColumnNames} from '../model/column-names';
+import {ColumnNamesService} from './column-names.service';
+import {TableMetadata} from '../model/table-metadata';
+import {SaveSearch} from '../model/save-search';
+import {AlertsSearchResponse} from '../model/alerts-search-response';
+import {SearchRequest} from '../model/search-request';
+
+export class ElasticSearchLocalstorageImpl extends DataSource {
+
+  private defaultColumnMetadata = [
+new ColumnMetadata('_id', 'string'),
+new ColumnMetadata('timestamp', 'date'),
+new ColumnMetadata('source:type', 'string'),
+new ColumnMetadata('ip_src_addr', 'ip'),
+new ColumnMetadata('enrichments:geo:ip_dst_addr:country', 'string'),
+new ColumnMetadata('ip_dst_addr', 'ip'),
+new ColumnMetadata('host', 'string'),
+new ColumnMetadata('alert_status', 'string')
+  ];
+
+  getAlerts(searchRequest: SearchRequest): 
Observable {
+let url = '/search/*,-*kibana/_search';
+return this.http.post(url, searchRequest, new RequestOptions({headers: 
new Headers(this.defaultHeaders)}))
--- End diff --

I would expect there to be some kind of manipulation to the searchRequest 
sent to Elasticsearch.  For example, Elasticsearch expects the request to 
follow this structure:
```
{
  "query":
  {
"query_string":
{   
  "query": "some lucene query"
}
  }
}
```
but our search abstraction is simpler:
```
{
  "query": "some lucene query"
}
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r126532937
  
--- Diff: metron-interface/metron-alerts/src/app/service/data-source.ts ---
@@ -0,0 +1,62 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Observable} from 'rxjs/Rx';
+import {Injectable} from '@angular/core';
+import {Http} from '@angular/http';
+
+import {Alert} from '../model/alert';
+import {ColumnMetadata} from '../model/column-metadata';
+import {ColumnNames} from '../model/column-names';
+import {TableMetadata} from '../model/table-metadata';
+import {SaveSearch} from '../model/save-search';
+import {AlertsSearchResponse} from '../model/alerts-search-response';
+import {SearchRequest} from '../model/search-request';
+
+@Injectable()
+export abstract class DataSource {
+  defaultHeaders: {'Content-Type': 'application/json', 'X-Requested-With': 
'XMLHttpRequest'};
+
+  constructor(protected http: Http) {}
+
+  // Calls to fetch alerts
+  abstract getAlerts(searchRequest: SearchRequest): 
Observable
+  abstract getAlert(index: string, type: string, alertId: string): 
Observable
+  abstract updateAlertState(request: any): Observable<{}>
+
+  // Calls to fetch default alert table column names and all the field 
names across all indexes
+  abstract getDefaultAlertTableColumnNames(): Observable
+  abstract getAllFieldNames(): Observable
+
+  // Calls to rename field names and to fetch the renamed field names
+  abstract getAlertTableColumnNames(): Observable
+  abstract saveAlertTableColumnNames(columns: ColumnNames[]): 
Observable<{}>
+
+  // Calls to fetch and save alerts table settings like refresh interval, 
page size, default selected table column names
+  abstract getAlertTableSettings(): Observable
--- End diff --

This function (and others in this class) really have nothing to do with a 
search engine or Elasticsearch.  I think would make sense to move these to a 
different abstract class.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r126532195
  
--- Diff: metron-interface/metron-alerts/src/app/model/search-request.ts ---
@@ -0,0 +1,7 @@
+export class SearchRequest {
+  _source: string[];
+  query = { query_string: { query: '' } };
--- End diff --

Shouldn't this type by 'string'?  This doesn't match the model in the PR 
comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r126531609
  
--- Diff: 
metron-interface/metron-alerts/src/app/alerts/saved-searches/saved-searches.component.ts
 ---
@@ -0,0 +1,124 @@
+import { Component, OnInit } from '@angular/core';
+import {Router} from '@angular/router';
+import {Observable} from 'rxjs/Rx';
+
+import {SaveSearchService} from '../../service/save-search.service';
+import {SaveSearch} from '../../model/save-search';
+import {MetronDialogBox} from '../../shared/metron-dialog-box';
+import {NUM_SAVED_SEARCH} from '../../utils/constants';
+
+@Component({
+  selector: 'app-saved-searches',
+  templateUrl: './saved-searches.component.html',
+  styleUrls: ['./saved-searches.component.scss']
+})
+export class SavedSearchesComponent implements OnInit {
+
+  searches: SaveSearch[];
+  recentSearcheObj: SaveSearch[];
+  savedSearches: any = {};
--- End diff --

Should these types by SaveSearch[]?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r126531076
  
--- Diff: 
metron-interface/metron-alerts/src/app/alerts/alerts-list/query-builder.ts ---
@@ -0,0 +1,139 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Filter} from '../../model/filter';
+import {ColumnNamesService} from '../../service/column-names.service';
+import {SearchRequest} from '../../model/search-request';
+
+export class QueryBuilder {
+  private _searchRequest = new SearchRequest();
+  private _query = '*';
+  private _displayQuery = this._query;
+  private _filters: Filter[] = [];
+
+  set query(value: string) {
+value = value.replace(/\\:/g, ':');
+this._query = value;
+this.updateFilters(this._query, false);
+this.onSearchChange();
+  }
+
+  get query(): string {
+return this._query;
+  }
+
+  set displayQuery(value: string) {
+this._displayQuery = value;
+this.updateFilters(this._displayQuery, true);
+this.onSearchChange();
+  }
+
+  get displayQuery(): string {
+return this._displayQuery;
+  }
+
+  get filters(): Filter[] {
+return this._filters;
+  }
+
+
+  get searchRequest(): SearchRequest {
+this._searchRequest.query = { query_string: { query: 
this.generateSelect() } };
+return this._searchRequest;
+  }
+
+  set searchRequest(value: SearchRequest) {
+this._searchRequest = value;
+this.query = this._searchRequest.query.query_string.query;
--- End diff --

Same as above.  I would expect:
`this.query = this._searchRequest.query`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r126530918
  
--- Diff: 
metron-interface/metron-alerts/src/app/alerts/alerts-list/query-builder.ts ---
@@ -0,0 +1,139 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Filter} from '../../model/filter';
+import {ColumnNamesService} from '../../service/column-names.service';
+import {SearchRequest} from '../../model/search-request';
+
+export class QueryBuilder {
+  private _searchRequest = new SearchRequest();
+  private _query = '*';
+  private _displayQuery = this._query;
+  private _filters: Filter[] = [];
+
+  set query(value: string) {
+value = value.replace(/\\:/g, ':');
+this._query = value;
+this.updateFilters(this._query, false);
+this.onSearchChange();
+  }
+
+  get query(): string {
+return this._query;
+  }
+
+  set displayQuery(value: string) {
+this._displayQuery = value;
+this.updateFilters(this._displayQuery, true);
+this.onSearchChange();
+  }
+
+  get displayQuery(): string {
+return this._displayQuery;
+  }
+
+  get filters(): Filter[] {
+return this._filters;
+  }
+
+
+  get searchRequest(): SearchRequest {
+this._searchRequest.query = { query_string: { query: 
this.generateSelect() } };
--- End diff --

This looks specific to Elasticsearch.  I would expect:
`this._searchRequest.query = this.generateSelect()`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #636: METRON-1022: Elasticsearch REST endpoint

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/636
  
If the es or solr code was packaged as a bundle ( it with it's dependencies 
-> the es and solr libs ), the bundle system let you load each implementation 
as a plugin, and will create a new classloader for each.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[jjmeyer0]

Github user jjmeyer0 commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126524234
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java
 ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+  final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+  @Test(expected = IllegalArgumentException.class)
+  public void nullArgumentListShouldThrowException() throws Exception {
+hash.apply(null);
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void emptyArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.emptyList());
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void singleArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.singletonList("some value."));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException3() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException4() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3", "4"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void invalidAlgorithmArgumentShouldThrowException() throws 
Exception {
+hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+  }
+
+  @Test
+  public void invalidNullAlgorithmArgumentShouldThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList("value to hash", null)));
+  }
+
+  @Test
+  public void nullInputForValueToHashShouldProperlyThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList(null, "md5")));
+  }
+
+  @Test
+  public void allAlgorithmsForMessageDigestShouldBeAbleToHash() throws 
Exception {
+final String valueToHash = "My value to hash";
+final Set algorithms = Security.getAlgorithms("MessageDigest");
+
+algorithms.forEach(algorithm -> {
+  try {
+final MessageDigest m = MessageDigest.getInstance(algorithm);
+m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+assertEquals(BaseEncoding.base16().encode(m.digest()), 
hash.apply(Arrays.asList(valueToHash, algorithm)));
+  } catch (NoSuchAlgorithmException e) {
+throw new RuntimeException(e);
--- End diff --

@mattf-horton since I'm using a lambda, I don't believe this is possible 
since `NoSuchAlgorithmException` is a checked exception.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #636: METRON-1022: Elasticsearch REST endpoint

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/636
  
The core issue is that both Solr and Elasticsearch depend on Lucene.  Our 
version of Elasticsearch (2.3.3) is old so there are lots of Lucene class 
version conflicts when adding Solr as a dependency.  We have the same problem 
with Guava because some libraries we depend on use old versions of Guava.  The 
fact that Guava aggressively deprecates it's API makes it challenging.  We 
solved this by creating a "shaded" ES module that shades Guava so that it 
doesn't conflict with the version of Guava used by other libraries.

I'm not that familiar with bundle loading.  How would we use that here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #644: METRON-1027: Errant Log Message When No GeoIP Hit on Vali...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/644
  
Oh, nevermind.  Got it.  Sorry, Travis.  Thank you for your service.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #644: METRON-1027: Errant Log Message When No GeoIP Hit ...

[nickwallen]

GitHub user nickwallen reopened a pull request:

https://github.com/apache/metron/pull/644

METRON-1027: Errant Log Message When No GeoIP Hit on Valid, External IPv4 
Address

When a valid public IPv4 address does not match a record in the Maxmind 
GeoIP database, a log message indicates that an error occurred with the 
GeoLite2 DB.  

```
2017-07-05 12:50:34.266 o.a.m.e.a.g.GeoLiteDatabase WARN Metron GeoLite2 DB 
encountered an error
com.maxmind.geoip2.exception.AddressNotFoundException: The address X.X.X.X 
is not in the database.
at com.maxmind.geoip2.DatabaseReader.get(DatabaseReader.java:157) 
~[stormjar.jar:?]
at com.maxmind.geoip2.DatabaseReader.city(DatabaseReader.java:202) 
~[stormjar.jar:?]
at 
org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase.get(GeoLiteDatabase.java:136)
 [stormjar.jar:?]
at 
org.apache.metron.enrichment.adapters.geo.GeoAdapter.enrich(GeoAdapter.java:47) 
[stormjar.jar:?]
at 
org.apache.metron.enrichment.adapters.geo.GeoAdapter.enrich(GeoAdapter.java:31) 
[stormjar.jar:?]
```

Really no error has occurred with the GeoIP database.  Not having a hit in 
the database is an expected condition and processing should continue without 
the errant log message.

### Testing

To test this change, setup a Metron environment with GeoIP enrichment 
enabled.  Ingest some telemetry containing an valid, external IPv4 address that 
is not contained within the GeoIP database.   You can use an IP like 
`203.0.113.1` as the range 203.0.113.0/24 is reserved for "TEST-NET-3" and 
should never be locatable.  Before this PR, you will see the noted error 
message in the Enrichment topology logs.  With this change, you will not see 
the error message in your logs.

## Pull Request Checklist

- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron 
- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1027

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/644.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #644


commit f4c669d1c5c8ab2e071f46531941d2016bf0995a
Author: Nick Allen 
Date:   2017-07-10T18:41:34Z

METRON-1027: Errant Log Message When No GeoIP Hit on Valid, External IPv4 
Address

commit 12e434e05c27a45961fd1ad7b6dfa711345904cc
Author: Nick Allen 
Date:   2017-07-10T18:59:51Z

AddressNotFoundException is an expected condition and should only log at 
debug level

commit e66f9761300462183a9aee360c891a507be2af1f
Author: Nick Allen 
Date:   2017-07-10T19:19:34Z

A lookup of private address should have an empty result




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #644: METRON-1027: Errant Log Message When No GeoIP Hit on Vali...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/644
  
Thanks for the review guys.  Just trying to understand why Travis is 
reporting a unit test failure.  Its like it is not running the latest code.  
Bah.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #636: METRON-1022: Elasticsearch REST endpoint

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/636
  
What kind of isolation we do need?  The 777 introduces bundle loading gives 
classloader isolation
If they were loaded as plugins then it may work.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #644: METRON-1027: Errant Log Message When No GeoIP Hit ...

[nickwallen]

Github user nickwallen closed the pull request at:

https://github.com/apache/metron/pull/644


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #636: METRON-1022: Elasticsearch REST endpoint

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/636
  
@ottobackwards just created a discuss thread on the general approach here.

@cestella you're absolutely correct.  We need a way for solr and ES to 
coexist.  A precedent has already been set in our indexing module but just want 
to throw another suggestion out there.  The downside of having separate modules 
that extend a common module is that automated deployment through Ambari MPack 
is more complex.  In fact, the current MPack doesn't support this right now.  

Ideally we could leverage Spring to easily swap out the implementation at 
runtime but we would need both Solr and ES classes available on the classpath.  
Would it be possible to add lucene classes (and possibly others) to the list of 
shaded classes in the elasticsearch-shaded module and create a similar module 
for Solr?  Or are there challenges I'm not thinking of and/or would it make 
things more confusing having different strategies in different parts of Metron?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #644: METRON-1027: Errant Log Message When No GeoIP Hit on Vali...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/644
  
+1 by inspection.  @nickwallen is right that the log already occurs in the 
GeoLiteDatabase, so I'm okay with dropping it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #636: METRON-1022: Elasticsearch REST endpoint

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/636#discussion_r126511864
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/writer/ElasticsearchWriter.java
 ---
@@ -60,96 +61,8 @@ public ElasticsearchWriter 
withOptionalSettings(Map optionalSett
   @Override
   public void init(Map stormConf, TopologyContext topologyContext, 
WriterConfiguration configurations) {
 Map globalConfiguration = 
configurations.getGlobalConfig();
--- End diff --

Agreed!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #636: METRON-1022: Elasticsearch REST endpoint

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/636#discussion_r126511838
  
--- Diff: 
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/ElasticsearchServiceImpl.java
 ---
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.rest.service.impl;
+
+import org.apache.metron.rest.RestException;
+import org.apache.metron.rest.model.SearchRequest;
+import org.apache.metron.rest.model.SearchResponse;
+import org.apache.metron.rest.model.SearchResult;
+import org.apache.metron.rest.model.SortField;
+import org.apache.metron.rest.service.SearchService;
+import org.elasticsearch.client.transport.TransportClient;
+import org.elasticsearch.index.query.QueryStringQueryBuilder;
+import org.elasticsearch.search.builder.SearchSourceBuilder;
+import org.elasticsearch.search.sort.FieldSortBuilder;
+import org.elasticsearch.search.sort.SortOrder;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Service;
+
+import java.util.Arrays;
+import java.util.stream.Collectors;
+
+@Service
+public class ElasticsearchServiceImpl implements SearchService {
+
+  private TransportClient client;
+
+  @Autowired
+  public ElasticsearchServiceImpl(TransportClient client) {
+this.client = client;
+  }
+
+  @Override
+  public SearchResponse search(SearchRequest searchRequest) throws 
RestException {
+SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder()
+.size(searchRequest.getSize())
+.from(searchRequest.getFrom())
+.query(new QueryStringQueryBuilder(searchRequest.getQuery()))
+.fetchSource(true)
+.trackScores(true);
+for(SortField sortField: searchRequest.getSort()) {
+  FieldSortBuilder fieldSortBuilder = new 
FieldSortBuilder(sortField.getField());
+  fieldSortBuilder.order(sortField.getSortOrder() == 
org.apache.metron.rest.model.SortOrder.DESC ? SortOrder.DESC : SortOrder.ASC);
+  searchSourceBuilder = searchSourceBuilder.sort(fieldSortBuilder);
+}
--- End diff --

This is a great suggestion.  We already have paging, just need to enforce 
page size and provide a sensible default.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[DISCUSS] Search in REST

[Ryan Merriman]

This discussion is an attempt to clarify some questions and discuss design
decisions related to METRON-1022.

The primary purpose of METRON-1022 is to provide a foundation for building
Metron-specific Elasticsearch (or other search engine implementations)
functions in our REST application.  This translates into 3 features
provided by METRON-1022:  a common approach to setting up a
TransportClient, a search abstraction layer and a simple Elasticsearch
implementation consisting of a single function.  I believe the setup part
is fairly straightforward and doesn't require a detailed discussion.
Please chime in if I'm wrong there.

The first order of business is to all agree on an architectural approach.
How and where should we query Elasticsearch?  METRON-1022 duplicates some
functionality in METRON-990 but is architecturally different.  Instead of
the client-side code interacting directly with Elasticsearch through it's
REST api, this PR interacts with Elasticsearch through the Java api in a
Metron REST service.  I believe there are a couple of advantages to doing
it this way:

   - A Metron-specific search service in REST can be reused by other UIs
   and clients.  It would be possible to make an angular service reusable but
   that would take some work and it would only be reusable in javascript as an
   imported library and not as flexible as a service available over http.
   - Metron provides an integration testing framework for Java-based
   classes.  METRON-1022 leverages this without much additional effort.  It
   would take a lot more work to enable javascript modules to use this.
   - In my experience, the Metron community is much more comfortable with
   developing and reviewing features written in Java as opposed to
   javascript.  I think that is important for foundational pieces like this.

Some arguments to consider for keeping Elasticsearch functions in a
javascript service:

   - More efficient since there is no proxy in the middle (Metron REST
   being the proxy)
   - Eliminates the task of resolving version conflicts that comes with
   adding the Elasticsearch dependency to a Maven module although there are
   ways to make this easier

The second topic to discuss is the search abstraction.  This has been
requested several times and I think there is consensus that we need it.
METRON-1022 attempts to do this by:

   - creating model classes that represent search requests/responses
   - creating a search interface that accepts these model classes as input
   and return parameters
   - creating a controller that exposes this interface over REST
   - using Spring's IOC framework to select the correct implementation

An implementation of a search function was included in METRON-1022 as an
example.  ElasticsearchServiceImpl implements SearchService and is selected
as the implementation by default.  This could have been a separate PR but I
felt having it in context would help reviewers understand the design
pattern.

How does this relate to METRON-990?  Currently they overlap with
METRON-1022 offering a subset of the functionality in the
METRON-990 Elasticsearch service.  The idea is to first ensure METRON-990
and METRON-1022 both conform to the same search abstraction (which has been
discussed in METRON-990 feedback).  The next step would be to replace the
search service in METRON-990 to one that queries the Metron REST service
instead.  Ideally this only involves changing one class since the
abstraction is used in all the other components of METRON-990 and is
trivial since the complexity is now in Metron REST and not javascript.
Next other services (getting index metadata for example) would be converted
using the same process in incremental PRs.  Then, moving forward, all
Elasticsearch interactions would instead be developed as Metron REST
endpoints using the foundation established in METRON-1022.

This is a lot to digest so I'm happy to more detail as needed.  Interested
to hear others' thoughts and reactions.

Ryan

[GitHub] metron pull request #644: METRON-1027: Errant Log Message When No GeoIP Hit ...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/644#discussion_r126510446
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/GeoAdapter.java
 ---
@@ -46,7 +46,6 @@ public JSONObject enrich(CacheKey value) {
 JSONObject enriched = new JSONObject();
 Optional> result = 
GeoLiteDatabase.INSTANCE.get(value.coerceValue(String.class));
 if(!result.isPresent()) {
-  _LOG.error("GEO Enrichment failure: {}", 
value.coerceValue(String.class));
--- End diff --

We already get logging in this case from GeoLiteDatabase:167.  I didn't 
think we needed another.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #644: METRON-1027: Errant Log Message When No GeoIP Hit on Vali...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/644
  
+1 by inspection.  I agree that the log should be moved in level and not 
dropped


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #644: METRON-1027: Errant Log Message When No GeoIP Hit on Vali...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/644
  
Specifically, I'd say add it in 
```
if (isIneligibleAddress(ip, addr)) {
  return Optional.of(new HashMap());
}
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #644: METRON-1027: Errant Log Message When No GeoIP Hit on Vali...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/644
  
@nickwallen Off the top of my head, I think it could be Optional.empty(), 
but I'd like to see a debug log added so we can easily tell the difference 
between "IP not found" and "It wasn't even something we should have looked up". 
 Double check the logic on the function, but it should essentially be the same 
thing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #644: METRON-1027: Errant Log Message When No GeoIP Hit ...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/644#discussion_r126507756
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/GeoAdapter.java
 ---
@@ -46,7 +46,6 @@ public JSONObject enrich(CacheKey value) {
 JSONObject enriched = new JSONObject();
 Optional> result = 
GeoLiteDatabase.INSTANCE.get(value.coerceValue(String.class));
 if(!result.isPresent()) {
-  _LOG.error("GEO Enrichment failure: {}", 
value.coerceValue(String.class));
--- End diff --

Can you make it log.debug()?  I think it's still useful to be able to tell 
what happened, it just shouldn't be error.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #644: METRON-1027: Errant Log Message When No GeoIP Hit on Vali...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/644
  
@justinleet Another thing I noticed while in the code is that a lookup of a 
private IP address is expected to return an empty map.  Since we're using an 
Optional, wouldn't it make more sense for it to return `Optional.empty()`?  

The javadocs say that `Optional.empty()` indicates a lookup miss, not an 
empty map.  I could either make it return `Optional.empty()` or I could update 
the docs so that they match current behavior.  Let me know what you think.

Thanks

 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #644: METRON-1027: Errant Log Message When No GeoIP Hit ...

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/644

METRON-1027: Errant Log Message When No GeoIP Hit on Valid, External …

When a valid public IPv4 address does not match a record in the Maxmind 
GeoIP database, a log message indicates that an error occurred with the 
GeoLite2 DB.  

```
2017-07-05 12:50:34.266 o.a.m.e.a.g.GeoLiteDatabase WARN Metron GeoLite2 DB 
encountered an error
com.maxmind.geoip2.exception.AddressNotFoundException: The address X.X.X.X 
is not in the database.
at com.maxmind.geoip2.DatabaseReader.get(DatabaseReader.java:157) 
~[stormjar.jar:?]
at com.maxmind.geoip2.DatabaseReader.city(DatabaseReader.java:202) 
~[stormjar.jar:?]
at 
org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase.get(GeoLiteDatabase.java:136)
 [stormjar.jar:?]
at 
org.apache.metron.enrichment.adapters.geo.GeoAdapter.enrich(GeoAdapter.java:47) 
[stormjar.jar:?]
at 
org.apache.metron.enrichment.adapters.geo.GeoAdapter.enrich(GeoAdapter.java:31) 
[stormjar.jar:?]
```

Really no error has occurred with the GeoIP database.  Not having a hit in 
the database is an expected condition and processing should continue without 
the errant log message.

### Testing

To test this change, setup a Metron environment with GeoIP enrichment 
enabled.  Ingest some telemetry containing an valid, external IPv4 address that 
is not contained within the GeoIP database.   For example `203.0.113.1` works  
as the range 203.0.113.0/24 is assigned as "TEST-NET-3" and should never be 
locatable.  Before this PR, you will see the error message in the Storm logs 
for the Enrichment topology.  With this change, you will not see the error 
message in your logs.

## Pull Request Checklist

- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron 
- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1027

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/644.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #644


commit f4c669d1c5c8ab2e071f46531941d2016bf0995a
Author: Nick Allen 
Date:   2017-07-10T18:41:34Z

METRON-1027: Errant Log Message When No GeoIP Hit on Valid, External IPv4 
Address




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #639: METRON-1013 add command line verification to stellar shel...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/639
  
I don't know what is up with the ProfilerIntegrationTest.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #639: METRON-1013 add command line verification to stell...

[ottobackwards]

GitHub user ottobackwards reopened a pull request:

https://github.com/apache/metron/pull/639

METRON-1013 add command line verification to stellar shell

Stellar does no verification of parameters passed on the command line.
It should check if the files don't exist for -v, -p, irc, or the zookeeper 
url fragment is malformed.

This PR adds that verification, and tests for those parameters


## Testing
* build and tests should run
* run stellar from the code base ( out of metron-stellar/stellar-common), 
passing in invalid arguments ( see the unit test for examples )
```bash
 » mvn exec:java \
   -Dexec.mainClass="org.apache.metron.stellar.common.shell.StellarShell" 
-Dexec.args="-z node1"
[INFO] Scanning for projects...
[INFO]
[INFO] 

[INFO] Building stellar-common 0.4.1
[INFO] 

[INFO]
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ stellar-common ---
Zookeeper option must have port: node1
```

* or run stellar from a metron install


## Pull Request Checklist

### For all changes:
- [x ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron stellar_verify_zookeeper

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/639.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #639


commit 47579a1bfba0f06def54d39b23c80646511f5a87
Author: Otto Fowler 
Date:   2017-07-08T16:03:40Z

add command line verification to stellar shell

commit 5abe8acb474807acf0027dd7f06d698ad2b4cb51
Author: Otto Fowler 
Date:   2017-07-08T16:14:11Z

do not output the exception, just the message to console

commit 18ef06e32024366920db9f99e2b297adbca1c3f0
Author: Otto Fowler 
Date:   2017-07-08T16:37:13Z

thought the version was the same

commit 6b64573b118a6927993a61848403630e3107a0de
Author: Otto Fowler 
Date:   2017-07-09T14:42:12Z

add port range checking

reformat vs. google style

address checkstyle




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #639: METRON-1013 add command line verification to stell...

[ottobackwards]

Github user ottobackwards closed the pull request at:

https://github.com/apache/metron/pull/639


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126474912
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+  @Stellar(
+name = "HASH",
+description = "Hashes a given value using the given hashing algorithm 
and returns a hex encoded string. This function only hashes " +
+  "strings and values that implement java.io.Serializable.",
+params = {
+  "toHash - value to hash.",
+  "hashType - A valid string representation of an algorithm supported 
by java.security.MessageDigest.",
+},
+returns = "A hex representation of a hashed value using the given 
hashing algorithm. If either argument is " +
+  "null then null will be returned. If the type of 'toHash' is neither 
a string nor of type java.io.Serializable, " +
+  "then null is returned."
+  )
+  public static class Hash extends BaseStellarFunction {
--- End diff --

Nice use of general Java hash algorithm i/f.  Suggest you also add a 
convenience Stellar function to return the available list of hash algs in 
current environment.  Perhaps GET_HASHES_AVAILABLE() -> List


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126481330
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java
 ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+  final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+  @Test(expected = IllegalArgumentException.class)
+  public void nullArgumentListShouldThrowException() throws Exception {
+hash.apply(null);
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void emptyArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.emptyList());
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void singleArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.singletonList("some value."));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException3() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException4() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3", "4"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void invalidAlgorithmArgumentShouldThrowException() throws 
Exception {
+hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+  }
+
+  @Test
+  public void invalidNullAlgorithmArgumentShouldThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList("value to hash", null)));
+  }
+
+  @Test
+  public void nullInputForValueToHashShouldProperlyThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList(null, "md5")));
+  }
+
+  @Test
+  public void allAlgorithmsForMessageDigestShouldBeAbleToHash() throws 
Exception {
+final String valueToHash = "My value to hash";
+final Set algorithms = Security.getAlgorithms("MessageDigest");
+
+algorithms.forEach(algorithm -> {
+  try {
+final MessageDigest m = MessageDigest.getInstance(algorithm);
+m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+assertEquals(BaseEncoding.base16().encode(m.digest()), 
hash.apply(Arrays.asList(valueToHash, algorithm)));
+  } catch (NoSuchAlgorithmException e) {
+throw new RuntimeException(e);
+  }
+});
+  }
+
+  @Test
+  public void 
allAlgorithmsForMessageDigestShouldBeAbleToHashDirectStellarCall() throws 
Exception {
+final String valueToHash = "My value to hash";
+final Set algorithms = Security.getAlgorithms("MessageDigest");
+
+algorithms.forEach(algorithm -> {
+  try {
+final Object actual = run("HASH('" + valueToHash + "', '" + 
algorithm + "')", Collections.emptyMap());
+
+final MessageDigest m = MessageDigest.getInstance(algorithm);
+m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+assertEquals(BaseEncoding.base

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126475943
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java
 ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+  final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+  @Test(expected = IllegalArgumentException.class)
+  public void nullArgumentListShouldThrowException() throws Exception {
+hash.apply(null);
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void emptyArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.emptyList());
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void singleArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.singletonList("some value."));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException3() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException4() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3", "4"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void invalidAlgorithmArgumentShouldThrowException() throws 
Exception {
+hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+  }
+
+  @Test
+  public void invalidNullAlgorithmArgumentShouldThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList("value to hash", null)));
+  }
+
+  @Test
+  public void nullInputForValueToHashShouldProperlyThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList(null, "md5")));
+  }
+
+  @Test
+  public void allAlgorithmsForMessageDigestShouldBeAbleToHash() throws 
Exception {
+final String valueToHash = "My value to hash";
+final Set algorithms = Security.getAlgorithms("MessageDigest");
+
+algorithms.forEach(algorithm -> {
+  try {
+final MessageDigest m = MessageDigest.getInstance(algorithm);
+m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+assertEquals(BaseEncoding.base16().encode(m.digest()), 
hash.apply(Arrays.asList(valueToHash, algorithm)));
+  } catch (NoSuchAlgorithmException e) {
+throw new RuntimeException(e);
--- End diff --

Unless there's some junit subtlety that isn't occurring to me right now, it 
would be cleaner to declare this testcase as "throws NoSuchAlgorithmException", 
and then there's no need for the try/catch context.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126476908
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java
 ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+  final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+  @Test(expected = IllegalArgumentException.class)
+  public void nullArgumentListShouldThrowException() throws Exception {
+hash.apply(null);
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void emptyArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.emptyList());
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void singleArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.singletonList("some value."));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException3() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException4() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3", "4"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void invalidAlgorithmArgumentShouldThrowException() throws 
Exception {
+hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+  }
+
+  @Test
+  public void invalidNullAlgorithmArgumentShouldThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList("value to hash", null)));
+  }
+
+  @Test
+  public void nullInputForValueToHashShouldProperlyThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList(null, "md5")));
+  }
+
+  @Test
+  public void allAlgorithmsForMessageDigestShouldBeAbleToHash() throws 
Exception {
+final String valueToHash = "My value to hash";
+final Set algorithms = Security.getAlgorithms("MessageDigest");
+
+algorithms.forEach(algorithm -> {
+  try {
+final MessageDigest m = MessageDigest.getInstance(algorithm);
+m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+assertEquals(BaseEncoding.base16().encode(m.digest()), 
hash.apply(Arrays.asList(valueToHash, algorithm)));
+  } catch (NoSuchAlgorithmException e) {
+throw new RuntimeException(e);
+  }
+});
+  }
+
+  @Test
+  public void 
allAlgorithmsForMessageDigestShouldBeAbleToHashDirectStellarCall() throws 
Exception {
+final String valueToHash = "My value to hash";
+final Set algorithms = Security.getAlgorithms("MessageDigest");
+
+algorithms.forEach(algorithm -> {
+  try {
+final Object actual = run("HASH('" + valueToHash + "', '" + 
algorithm + "')", Collections.emptyMap());
+
+final MessageDigest m = MessageDigest.getInstance(algorithm);
--- End diff --

Nit: call this "expected" instead of "m" would read more clearly, here and 
in each test 

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126474167
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java
 ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+  final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+  @Test(expected = IllegalArgumentException.class)
+  public void nullArgumentListShouldThrowException() throws Exception {
+hash.apply(null);
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void emptyArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.emptyList());
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void singleArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.singletonList("some value."));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException3() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException4() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3", "4"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void invalidAlgorithmArgumentShouldThrowException() throws 
Exception {
+hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+  }
+
+  @Test
+  public void invalidNullAlgorithmArgumentShouldThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList("value to hash", null)));
+  }
+
+  @Test
+  public void nullInputForValueToHashShouldProperlyThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList(null, "md5")));
+  }
--- End diff --

Not sure I agree about this one.  There's a lot of cases, including Stellar 
validation, where functions can get fed null arguments when you might not 
expect it.  What if we establish a norm that hash of null returns 0x00, for all 
hashing algorithms?

If you leave this one as is, remove "Properly" from testcase name and add 
"@Test(expected = IllegalArgumentException.class)".


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126473615
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java
 ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+  final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+  @Test(expected = IllegalArgumentException.class)
+  public void nullArgumentListShouldThrowException() throws Exception {
+hash.apply(null);
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void emptyArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.emptyList());
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void singleArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.singletonList("some value."));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException3() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException4() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3", "4"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void invalidAlgorithmArgumentShouldThrowException() throws 
Exception {
+hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+  }
+
+  @Test
+  public void invalidNullAlgorithmArgumentShouldThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList("value to hash", null)));
+  }
--- End diff --

Need "@Test(expected = IllegalArgumentException.class)" on this one too, 
don't we?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: threatintel_taxii_load.sh throws exception

[Casey Stella]

Thanks Vladimir, https://github.com/apache/metron/pull/643 was submitted a
few moments ago.

On Mon, Jul 10, 2017 at 3:35 PM, Vladimir Shlyakhtin <
vladimir.shlyakh...@sstech.us> wrote:

> Filed METRON-1026
>
> - Vladimir
> 
> From: Otto Fowler [ottobackwa...@gmail.com]
> Sent: Monday, July 10, 2017 9:13 AM
> To: dev@metron.apache.org; Vladimir Shlyakhtin
> Subject: Re: threatintel_taxii_load.sh throws exception
>
> After upgrading to 0.4.0?  I am not sure you should be running the
> threatintell_taxii_load.sh out of /usr/metron/0.3.1.
> What happens when you run it out of /usr/metron/0.4.1 ?
>
>
>
>
> On July 10, 2017 at 08:19:18, Vladimir Shlyakhtin (
> vladimir.shlyakh...@sstech.us)
> wrote:
>
> Hello,
>
> After upgrading from 0.3.0 version we noticed that taxii loader does not
> work.
>
> Here is details:
>
> # /usr/metron/0.3.1/bin/threatintel_taxii_load.sh -c
> /usr/local/opentaxii/etc/connections.conf/guest.phishtank_com.json -e
> /usr/local/opentaxii/etc/connections.conf/extractor.json -p 1
> 17/07/10 07:35:42 WARN extractor.TransformFilterExtractorDecorator:
> Unable to setup zookeeper client - zk_quorum url not provided. **This will
> limit some Stellar functionality**
> Exception in thread "main" java.lang.IllegalStateException: Extractor
> must be a STIX Extractor
> at org.apache.metron.dataloads.nonbulk.taxii.TaxiiLoader.
> main(TaxiiLoader.java:202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
>
>
> # cat /usr/local/opentaxii/etc/connections.conf/guest.phishtank_com.json
> {
> "endpoint" : "http://10.10.110.23:9000/services/discovery";
> ,"port" : "9000"
> ,"type" : "DISCOVER"
> ,"collection" : "guest.phishtank_com"
> ,"table" : "threatintel"
> ,"columnFamily" : "t"
> ,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ]
> }
>
> # cat /usr/local/opentaxii/etc/connections.conf/extractor.json
> {
> "config": {
> "columns": {
> "domain": 0
> },
> "indicator_column": "domain",
> "type" : "malicious_domain",
> "separator" : ","
> },
> "extractor" : "STIX"
> }
>
>
> "zk_quorum" parameter we specified as well and got exception (without
> "Unable to setup zookeeper client" warning).
>
> Exception is thrown due to false in condition:
> ```
> if(e instanceof StixExtractor) {
> ```
>
> in file TaxiiLoader.java.
>
> I changed exception message to:
> ```
> throw new IllegalStateException("Extractor must be a STIX Extractor" + "
> - " + e.getClass().getName());
> ```
>
> and got:
> ```
> Extractor must be a STIX Extractor - org.apache.metron.dataloads.
> extractor.TransformFilterExtractorDecorator
> ```
>
> Seems like it is related to commit c5bbf5acef05de16a18db9397745a6
> 795427f6b8.
>
> Let me know if this is bug and I should open jira issue.
>
> Thank you
>
>
> - Vladimir
>

[GitHub] metron pull request #643: METRON-1026: threatintel_taxii_load.sh throws exce...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/643

METRON-1026: threatintel_taxii_load.sh throws exception

## Contributor Comments
The delegation wrapping done to enable Stellar transformations on data as 
it is imported broke the taxii loader.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron taxii_error

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/643.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #643






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

RE: threatintel_taxii_load.sh throws exception

[Vladimir Shlyakhtin]

Filed METRON-1026

- Vladimir

From: Otto Fowler [ottobackwa...@gmail.com]
Sent: Monday, July 10, 2017 9:13 AM
To: dev@metron.apache.org; Vladimir Shlyakhtin
Subject: Re: threatintel_taxii_load.sh throws exception

After upgrading to 0.4.0?  I am not sure you should be running the 
threatintell_taxii_load.sh out of /usr/metron/0.3.1.
What happens when you run it out of /usr/metron/0.4.1 ?




On July 10, 2017 at 08:19:18, Vladimir Shlyakhtin 
(vladimir.shlyakh...@sstech.us) wrote:

Hello,

After upgrading from 0.3.0 version we noticed that taxii loader does not work.

Here is details:

# /usr/metron/0.3.1/bin/threatintel_taxii_load.sh -c 
/usr/local/opentaxii/etc/connections.conf/guest.phishtank_com.json -e 
/usr/local/opentaxii/etc/connections.conf/extractor.json -p 1
17/07/10 07:35:42 WARN extractor.TransformFilterExtractorDecorator: Unable to 
setup zookeeper client - zk_quorum url not provided. **This will limit some 
Stellar functionality**
Exception in thread "main" java.lang.IllegalStateException: Extractor must be a 
STIX Extractor
at 
org.apache.metron.dataloads.nonbulk.taxii.TaxiiLoader.main(TaxiiLoader.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)


# cat /usr/local/opentaxii/etc/connections.conf/guest.phishtank_com.json
{
"endpoint" : "http://10.10.110.23:9000/services/discovery";
,"port" : "9000"
,"type" : "DISCOVER"
,"collection" : "guest.phishtank_com"
,"table" : "threatintel"
,"columnFamily" : "t"
,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ]
}

# cat /usr/local/opentaxii/etc/connections.conf/extractor.json
{
"config": {
"columns": {
"domain": 0
},
"indicator_column": "domain",
"type" : "malicious_domain",
"separator" : ","
},
"extractor" : "STIX"
}


"zk_quorum" parameter we specified as well and got exception (without "Unable 
to setup zookeeper client" warning).

Exception is thrown due to false in condition:
```
if(e instanceof StixExtractor) {
```

in file TaxiiLoader.java.

I changed exception message to:
```
throw new IllegalStateException("Extractor must be a STIX Extractor" + " - " + 
e.getClass().getName());
```

and got:
```
Extractor must be a STIX Extractor - 
org.apache.metron.dataloads.extractor.TransformFilterExtractorDecorator
```

Seems like it is related to commit c5bbf5acef05de16a18db9397745a6795427f6b8.

Let me know if this is bug and I should open jira issue.

Thank you


- Vladimir

[GitHub] metron pull request #642: METRON-984 Stellar functions to decode encoded fie...

[ottobackwards]

GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/642

METRON-984 Stellar functions to decode encoded fields or strings

This functionality involves ( as a start ) encodings based on the Apache 
Commons Codec library ( binary )

Support exists for the following encodings:
 * Base32
 * Base32 as hex
 * Base 64
 * Binary
 * Hex

The Stellar functions added:
* LIST_SUPPORTED_ENCODINGS : List the supported encodings
* IS_ENCODING : is this string encoded this way?
* DECODE : decode a string with a given encoding, optionally verify it is 
in fact encoded that way

## Contributor Comments
[Please place any comments here.  A description of the problem/enhancement, 
how to reproduce the issue, your testing methodology, etc.]

### Testing
- Build and tests should run
- run the stellar shell
```bash
cd metron/metron-stellar/stellar-common
mvn -DskipTests package && mvn exec:java 
-Dexec.mainClass="org.apache.metron.stellar.common.shell.StellarShell"
```
- execute the functions on some strings

> I think I will add in ENCODE as well, then you can round trip test

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [na] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron stellar_encoding

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/642.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #642


commit 633d314c1518c5907b7c96113595aafbae3b95d2
Author: Otto Fowler 
Date:   2017-07-09T18:17:15Z

Stellar encoding functionality

This functionality involves ( as a start ) encodings based on the Apache 
Commons Codec library ( binary )

Support exists for the following encodings:
 * Base32
 * Base32 as hex
 * Base 64
 * Binary
 * Hex

The Stellar functions added:
* LIST_SUPPORTED_ENCODINGS : List the supported encodings
* IS_ENCODING : is this string encoded this way?
* DECODE : decode a string with a given encoding, optionally verify it is 
in fact encoded that way




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[jjmeyer0]

GitHub user jjmeyer0 opened a pull request:

https://github.com/apache/metron/pull/641

METRON-539: added HASH function for stellar.

## Contributor Comments
Added a hash function to stellar. Many unit tests were created. Along with 
these tests I opened the Stellar terminal and did the following:

### Start Stellar
```bash
mvn exec:java 
-Dexec.mainClass="org.apache.metron.stellar.common.shell.StellarShell" -pl 
metron-stellar/stellar-common/
```

### Test `HASH` Function

Below is the output of some of the tests I ran. To get all available inputs 
for the second argument you can call `Security.getAlgorithms("MessageDigest")` 
and print out the array. However, 'SHA-1', 'MD5', and 'SHA-256' are required by 
all Java platforms. Please see 
[MessageDigest](https://docs.oracle.com/javase/8/docs/api/java/security/MessageDigest.html)
 and 
[Security](https://docs.oracle.com/javase/8/docs/api/java/security/Security.html).
 

```bash
[Stellar]>>> Functions loaded, you may refer to functions now...

[Stellar]>>> some_string := 'my string to hash'
[Stellar]>>> HASH(some_string, 'md5')
E08863A4BE232F166C350F45C9DAA586
[Stellar]>>> some_serializable_object := [ 1, 2, 3 ]
[Stellar]>>> HASH(some_serializable_object, 'MD5')
2383CD011F1DA06A173ADE2F3F99CF62
[Stellar]>>> 
```

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jjmeyer0/incubator-metron METRON-539

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/641.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #641


commit 8de6390825e3a17de226ace4764eb6a9d4cec609
Author: JJ 
Date:   2017-07-10T12:34:02Z

METRON-539: added HASH function for stellar.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ti

RE: threatintel_taxii_load.sh throws exception

[Vladimir Shlyakhtin]

We have both version installed (0.3.1 and 0.4.0), the same exception for both 
version.
The same for latest (0.4.1)

- Vladimir

From: Otto Fowler [ottobackwa...@gmail.com]
Sent: Monday, July 10, 2017 9:13 AM
To: dev@metron.apache.org; Vladimir Shlyakhtin
Subject: Re: threatintel_taxii_load.sh throws exception

After upgrading to 0.4.0?  I am not sure you should be running the 
threatintell_taxii_load.sh out of /usr/metron/0.3.1.
What happens when you run it out of /usr/metron/0.4.1 ?




On July 10, 2017 at 08:19:18, Vladimir Shlyakhtin 
(vladimir.shlyakh...@sstech.us) wrote:

Hello,

After upgrading from 0.3.0 version we noticed that taxii loader does not work.

Here is details:

# /usr/metron/0.3.1/bin/threatintel_taxii_load.sh -c 
/usr/local/opentaxii/etc/connections.conf/guest.phishtank_com.json -e 
/usr/local/opentaxii/etc/connections.conf/extractor.json -p 1
17/07/10 07:35:42 WARN extractor.TransformFilterExtractorDecorator: Unable to 
setup zookeeper client - zk_quorum url not provided. **This will limit some 
Stellar functionality**
Exception in thread "main" java.lang.IllegalStateException: Extractor must be a 
STIX Extractor
at 
org.apache.metron.dataloads.nonbulk.taxii.TaxiiLoader.main(TaxiiLoader.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)


# cat /usr/local/opentaxii/etc/connections.conf/guest.phishtank_com.json
{
"endpoint" : "http://10.10.110.23:9000/services/discovery";
,"port" : "9000"
,"type" : "DISCOVER"
,"collection" : "guest.phishtank_com"
,"table" : "threatintel"
,"columnFamily" : "t"
,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ]
}

# cat /usr/local/opentaxii/etc/connections.conf/extractor.json
{
"config": {
"columns": {
"domain": 0
},
"indicator_column": "domain",
"type" : "malicious_domain",
"separator" : ","
},
"extractor" : "STIX"
}


"zk_quorum" parameter we specified as well and got exception (without "Unable 
to setup zookeeper client" warning).

Exception is thrown due to false in condition:
```
if(e instanceof StixExtractor) {
```

in file TaxiiLoader.java.

I changed exception message to:
```
throw new IllegalStateException("Extractor must be a STIX Extractor" + " - " + 
e.getClass().getName());
```

and got:
```
Extractor must be a STIX Extractor - 
org.apache.metron.dataloads.extractor.TransformFilterExtractorDecorator
```

Seems like it is related to commit c5bbf5acef05de16a18db9397745a6795427f6b8.

Let me know if this is bug and I should open jira issue.

Thank you


- Vladimir

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

[iraghumitra]

Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/620
  
@mraliagha 
 - For 1 can you check with the latest code base. I am unable to simulate 
this. 
-  For 2 I am slightly confused here. I wanted to know the result of the 
GET rest call` 
http://:/_cluster/state`. 
This call fetches all the fields from elastic. All the default fields are added 
via Javascript so the rest call showing just the default fields is not 
possible. 
The rest call should either show all fields or show an error. Am I missing 
anything here?  You can fire a get call from browser to check the result.

Thanks for taking the time to review :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #636: METRON-1022: Elasticsearch REST endpoint

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/636
  
This is good work!  Thanks, Ryan.  We desperately need this abstraction.  I 
think the confusion may be that this PR is the general abstraction as well as a 
concrete implementation for one index.  I think, as we found with the indexing 
topology, that solr and ES won't be able to exist within the same project due 
to classpath issues (note we have 2 separate projects for solr and ES to 
isolate the dependencies completely).  I would think that the ES specific stuff 
may should be separated into a separate project (`metron-rest-elasticsearch`) 
where the abstraction should exist in `metron-rest`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: threatintel_taxii_load.sh throws exception

[Otto Fowler]

After upgrading to 0.4.0?  I am not sure you should be running the
threatintell_taxii_load.sh out of /usr/metron/0.3.1.
What happens when you run it out of /usr/metron/0.4.1 ?



On July 10, 2017 at 08:19:18, Vladimir Shlyakhtin (
vladimir.shlyakh...@sstech.us) wrote:

Hello,

After upgrading from 0.3.0 version we noticed that taxii loader does not
work.

Here is details:

# /usr/metron/0.3.1/bin/threatintel_taxii_load.sh -c
/usr/local/opentaxii/etc/connections.conf/guest.phishtank_com.json -e
/usr/local/opentaxii/etc/connections.conf/extractor.json -p 1
17/07/10 07:35:42 WARN extractor.TransformFilterExtractorDecorator: Unable
to setup zookeeper client - zk_quorum url not provided. **This will limit
some Stellar functionality**
Exception in thread "main" java.lang.IllegalStateException: Extractor must
be a STIX Extractor
at
org.apache.metron.dataloads.nonbulk.taxii.TaxiiLoader.main(TaxiiLoader.java:202)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)


# cat /usr/local/opentaxii/etc/connections.conf/guest.phishtank_com.json
{
"endpoint" : "http://10.10.110.23:9000/services/discovery";
,"port" : "9000"
,"type" : "DISCOVER"
,"collection" : "guest.phishtank_com"
,"table" : "threatintel"
,"columnFamily" : "t"
,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ]
}

# cat /usr/local/opentaxii/etc/connections.conf/extractor.json
{
"config": {
"columns": {
"domain": 0
},
"indicator_column": "domain",
"type" : "malicious_domain",
"separator" : ","
},
"extractor" : "STIX"
}


"zk_quorum" parameter we specified as well and got exception (without
"Unable to setup zookeeper client" warning).

Exception is thrown due to false in condition:
```
if(e instanceof StixExtractor) {
```

in file TaxiiLoader.java.

I changed exception message to:
```
throw new IllegalStateException("Extractor must be a STIX Extractor" + " -
" + e.getClass().getName());
```

and got:
```
Extractor must be a STIX Extractor -
org.apache.metron.dataloads.extractor.TransformFilterExtractorDecorator
```

Seems like it is related to commit
c5bbf5acef05de16a18db9397745a6795427f6b8.

Let me know if this is bug and I should open jira issue.

Thank you


- Vladimir

Re: threatintel_taxii_load.sh throws exception

[Casey Stella]

This is absolutely a bug and you should open a JIRA.

On Mon, Jul 10, 2017 at 1:11 PM, Vladimir Shlyakhtin <
vladimir.shlyakh...@sstech.us> wrote:

> Hello,
>
> After upgrading from 0.3.0 version we noticed that taxii loader does not
> work.
>
> Here is details:
>
> # /usr/metron/0.3.1/bin/threatintel_taxii_load.sh -c
> /usr/local/opentaxii/etc/connections.conf/guest.phishtank_com.json -e
> /usr/local/opentaxii/etc/connections.conf/extractor.json -p 1
> 17/07/10 07:35:42 WARN extractor.TransformFilterExtractorDecorator:
> Unable to setup zookeeper client - zk_quorum url not provided. **This will
> limit some Stellar functionality**
> Exception in thread "main" java.lang.IllegalStateException: Extractor
> must be a STIX Extractor
> at org.apache.metron.dataloads.nonbulk.taxii.TaxiiLoader.
> main(TaxiiLoader.java:202)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
>
>
> # cat /usr/local/opentaxii/etc/connections.conf/guest.phishtank_com.json
> {
>   "endpoint" : "http://10.10.110.23:9000/services/discovery";
>   ,"port" : "9000"
>   ,"type" : "DISCOVER"
>   ,"collection" : "guest.phishtank_com"
>   ,"table" : "threatintel"
>   ,"columnFamily" : "t"
>   ,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ]
> }
>
> # cat /usr/local/opentaxii/etc/connections.conf/extractor.json
> {
>   "config": {
> "columns": {
>   "domain": 0
> },
> "indicator_column": "domain",
> "type" : "malicious_domain",
> "separator" : ","
>   },
>   "extractor" : "STIX"
> }
>
>
> "zk_quorum"  parameter we specified as well and got exception (without
> "Unable to setup zookeeper client" warning).
>
> Exception is thrown due to false in condition:
> ```
> if(e instanceof StixExtractor) {
> ```
>
> in file TaxiiLoader.java.
>
> I changed exception message to:
> ```
> throw new IllegalStateException("Extractor must be a STIX Extractor" + "
> - " + e.getClass().getName());
> ```
>
> and got:
> ```
> Extractor must be a STIX Extractor - org.apache.metron.dataloads.
> extractor.TransformFilterExtractorDecorator
> ```
>
> Seems like it is related to commit c5bbf5acef05de16a18db9397745a6
> 795427f6b8.
>
> Let me know if this is bug and I should open jira issue.
>
> Thank you
>
>
> - Vladimir
>

Re: threatintel_taxii_load.sh throws exception

[Casey Stella]

Also, please reply here with the JIRA and I'll submit a PR.  It's a very
easy fix.

On Mon, Jul 10, 2017 at 1:24 PM, Casey Stella  wrote:

> This is absolutely a bug and you should open a JIRA.
>
> On Mon, Jul 10, 2017 at 1:11 PM, Vladimir Shlyakhtin <
> vladimir.shlyakh...@sstech.us> wrote:
>
>> Hello,
>>
>> After upgrading from 0.3.0 version we noticed that taxii loader does not
>> work.
>>
>> Here is details:
>>
>> # /usr/metron/0.3.1/bin/threatintel_taxii_load.sh -c
>> /usr/local/opentaxii/etc/connections.conf/guest.phishtank_com.json -e
>> /usr/local/opentaxii/etc/connections.conf/extractor.json -p 1
>> 17/07/10 07:35:42 WARN extractor.TransformFilterExtractorDecorator:
>> Unable to setup zookeeper client - zk_quorum url not provided. **This will
>> limit some Stellar functionality**
>> Exception in thread "main" java.lang.IllegalStateException: Extractor
>> must be a STIX Extractor
>> at org.apache.metron.dataloads.nonbulk.taxii.TaxiiLoader.main(
>> TaxiiLoader.java:202)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>> ssorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
>>
>>
>> # cat /usr/local/opentaxii/etc/connections.conf/guest.phishtank_com.json
>> {
>>   "endpoint" : "http://10.10.110.23:9000/services/discovery";
>>   ,"port" : "9000"
>>   ,"type" : "DISCOVER"
>>   ,"collection" : "guest.phishtank_com"
>>   ,"table" : "threatintel"
>>   ,"columnFamily" : "t"
>>   ,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ]
>> }
>>
>> # cat /usr/local/opentaxii/etc/connections.conf/extractor.json
>> {
>>   "config": {
>> "columns": {
>>   "domain": 0
>> },
>> "indicator_column": "domain",
>> "type" : "malicious_domain",
>> "separator" : ","
>>   },
>>   "extractor" : "STIX"
>> }
>>
>>
>> "zk_quorum"  parameter we specified as well and got exception (without
>> "Unable to setup zookeeper client" warning).
>>
>> Exception is thrown due to false in condition:
>> ```
>> if(e instanceof StixExtractor) {
>> ```
>>
>> in file TaxiiLoader.java.
>>
>> I changed exception message to:
>> ```
>> throw new IllegalStateException("Extractor must be a STIX Extractor" + "
>> - " + e.getClass().getName());
>> ```
>>
>> and got:
>> ```
>> Extractor must be a STIX Extractor - org.apache.metron.dataloads.ex
>> tractor.TransformFilterExtractorDecorator
>> ```
>>
>> Seems like it is related to commit c5bbf5acef05de16a18db9397745a6
>> 795427f6b8.
>>
>> Let me know if this is bug and I should open jira issue.
>>
>> Thank you
>>
>>
>> - Vladimir
>>
>
>

threatintel_taxii_load.sh throws exception

[Vladimir Shlyakhtin]

Hello,

After upgrading from 0.3.0 version we noticed that taxii loader does not work.

Here is details:

# /usr/metron/0.3.1/bin/threatintel_taxii_load.sh -c 
/usr/local/opentaxii/etc/connections.conf/guest.phishtank_com.json -e 
/usr/local/opentaxii/etc/connections.conf/extractor.json -p 1
17/07/10 07:35:42 WARN extractor.TransformFilterExtractorDecorator: Unable to 
setup zookeeper client - zk_quorum url not provided. **This will limit some 
Stellar functionality**
Exception in thread "main" java.lang.IllegalStateException: Extractor must be a 
STIX Extractor
at 
org.apache.metron.dataloads.nonbulk.taxii.TaxiiLoader.main(TaxiiLoader.java:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)


# cat /usr/local/opentaxii/etc/connections.conf/guest.phishtank_com.json
{
  "endpoint" : "http://10.10.110.23:9000/services/discovery";
  ,"port" : "9000"
  ,"type" : "DISCOVER"
  ,"collection" : "guest.phishtank_com"
  ,"table" : "threatintel"
  ,"columnFamily" : "t"
  ,"allowedIndicatorTypes" : [ "domainname:FQDN", "address:IPV_4_ADDR" ]
}

# cat /usr/local/opentaxii/etc/connections.conf/extractor.json
{
  "config": {
"columns": {
  "domain": 0
},
"indicator_column": "domain",
"type" : "malicious_domain",
"separator" : ","
  },
  "extractor" : "STIX"
}


"zk_quorum"  parameter we specified as well and got exception (without "Unable 
to setup zookeeper client" warning).

Exception is thrown due to false in condition:
```
if(e instanceof StixExtractor) {
```

in file TaxiiLoader.java.

I changed exception message to:
```
throw new IllegalStateException("Extractor must be a STIX Extractor" + " - " + 
e.getClass().getName());
```

and got:
```
Extractor must be a STIX Extractor - 
org.apache.metron.dataloads.extractor.TransformFilterExtractorDecorator
```

Seems like it is related to commit c5bbf5acef05de16a18db9397745a6795427f6b8.

Let me know if this is bug and I should open jira issue.

Thank you


- Vladimir

[GitHub] metron issue #520: METRON-833: Update MaaS documentation to explain how it i...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/520
  
+1, thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #520: METRON-833: Update MaaS documentation to explain how it i...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/520
  
Deconflicted; thanks for the patience.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #638: METRON-933 New stellar function for regex group capture

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/638
  
I love it.  The pattern cache is great!  +1 by inspection


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #621: METRON-1001: Allow metron to ingest parser metadat...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/621#discussion_r126377527
  
--- Diff: 
metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
 ---
@@ -176,16 +182,48 @@ public SimpleStormKafkaBuilder( Map 
kafkaProps
 , List fieldsConfiguration
 )
   {
+this(kafkaProps, toSubscription(topic), zkQuorum, fieldsConfiguration);
+  }
+
+  /**
+   * Create an object with the specified properties and exposing the 
specified fields.
+   * @param kafkaProps The special kafka properties
+   * @param subscription The subscription to the kafka topic(s)
+   * @param zkQuorum The zookeeper quorum.  We will use this to pull the 
brokers from this.
+   * @param fieldsConfiguration The fields to expose in the storm tuple 
emitted.
+   */
+  public SimpleStormKafkaBuilder( Map kafkaProps
+, Subscription subscription
+, String zkQuorum
+, List fieldsConfiguration
+)
+  {
 super( getBootstrapServers(zkQuorum, kafkaProps)
  , 
createDeserializer(Optional.ofNullable((String)kafkaProps.get(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG)),
 DEFAULT_DESERIALIZER)
  , 
createDeserializer(Optional.ofNullable((String)kafkaProps.get(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG)),
 DEFAULT_DESERIALIZER)
- , topic
+ , subscription
 );
 setProp(kafkaProps);
 setRecordTranslator(new 
SpoutRecordTranslator<>(FieldsConfiguration.toList(fieldsConfiguration)));
-this.topic = topic;
   }
 
+
+  private static Subscription toSubscription(String topicOrSubscription) {
+if (StringUtils.isEmpty(topicOrSubscription)) {
+  throw new IllegalStateException("Topic name is invalid: empty or 
null");
--- End diff --

Yep, done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---