[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126849833
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java
 ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+  final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+  @Test(expected = IllegalArgumentException.class)
+  public void nullArgumentListShouldThrowException() throws Exception {
+hash.apply(null);
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void emptyArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.emptyList());
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void singleArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.singletonList("some value."));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException3() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException4() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3", "4"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void invalidAlgorithmArgumentShouldThrowException() throws 
Exception {
+hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+  }
+
+  @Test
+  public void invalidNullAlgorithmArgumentShouldThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList("value to hash", null)));
+  }
+
+  @Test
+  public void nullInputForValueToHashShouldProperlyThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList(null, "md5")));
+  }
--- End diff --

Um, but if the hash values are always supposed to be of a certain length, 
then pad that out to as many zero digits as necessary to be conforming.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126849654
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java
 ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+  final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+  @Test(expected = IllegalArgumentException.class)
+  public void nullArgumentListShouldThrowException() throws Exception {
+hash.apply(null);
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void emptyArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.emptyList());
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void singleArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.singletonList("some value."));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException3() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException4() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3", "4"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void invalidAlgorithmArgumentShouldThrowException() throws 
Exception {
+hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+  }
+
+  @Test
+  public void invalidNullAlgorithmArgumentShouldThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList("value to hash", null)));
+  }
+
+  @Test
+  public void nullInputForValueToHashShouldProperlyThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList(null, "md5")));
+  }
--- End diff --

Ah, I was trusting your very detailed testcase names :-)  If they don't 
throw exceptions then of course the annotation should be consistent.

I remain of the opinion that it would be good for HASH to be able to return 
a conforming result for null objects, ie "00" instead of null.  But this is 
mostly an aesthetic judgment on my part, so I could be talked out of it.

You gave me a headache by prefixing the "00" string with the unicode symbol 
:-) and no, "\u" would not be the desirable return value; that would be the 
unicode codepoint for a single null character.  Yuck!  But I shouldn't have 
used the hex prefix "0x" either.  Since we are returning unadorned string 
values representing hex values, it should be "00" (a pair of zero digit 
characters in a String of length 2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126848506
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java
 ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+  final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+  @Test(expected = IllegalArgumentException.class)
+  public void nullArgumentListShouldThrowException() throws Exception {
+hash.apply(null);
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void emptyArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.emptyList());
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void singleArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.singletonList("some value."));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException3() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException4() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3", "4"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void invalidAlgorithmArgumentShouldThrowException() throws 
Exception {
+hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+  }
+
+  @Test
+  public void invalidNullAlgorithmArgumentShouldThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList("value to hash", null)));
+  }
+
+  @Test
+  public void nullInputForValueToHashShouldProperlyThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList(null, "md5")));
+  }
+
+  @Test
+  public void allAlgorithmsForMessageDigestShouldBeAbleToHash() throws 
Exception {
+final String valueToHash = "My value to hash";
+final Set algorithms = Security.getAlgorithms("MessageDigest");
+
+algorithms.forEach(algorithm -> {
+  try {
+final MessageDigest m = MessageDigest.getInstance(algorithm);
+m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+assertEquals(BaseEncoding.base16().encode(m.digest()), 
hash.apply(Arrays.asList(valueToHash, algorithm)));
+  } catch (NoSuchAlgorithmException e) {
+throw new RuntimeException(e);
+  }
+});
+  }
+
+  @Test
+  public void 
allAlgorithmsForMessageDigestShouldBeAbleToHashDirectStellarCall() throws 
Exception {
+final String valueToHash = "My value to hash";
+final Set algorithms = Security.getAlgorithms("MessageDigest");
+
+algorithms.forEach(algorithm -> {
+  try {
+final Object actual = run("HASH('" + valueToHash + "', '" + 
algorithm + "')", Collections.emptyMap());
+
+final MessageDigest m = MessageDigest.getInstance(algorithm);
+m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+

Re: [VOTE][PROPOSAL] minor changes to release process

[Matt Foley]

Hi, the below changes have been implemented.  I also wrote a new page, how to 
“Change the Build Version Number” (since that’s not so simple to do), and I 
moved “Website PR Merge” under “Release Process” in the wiki page hierarchy.

Please review.  If anyone feels I overstepped with editorial changes, please 
bring it up and we’ll correct.
Thanks,
--Matt

On 7/5/17, 2:47 PM, "Matt Foley"  wrote:

(The below proposal is also stated in 
https://issues.apache.org/jira/browse/METRON-1020 )

The following proposed changes are small, but not just editorial in nature, 
hence will require vote of the community to change. Our bylaws don’t have an 
action type of Modifying Policy, but it’s probably fair to consider policies to 
be “included by reference” in Bylaws, so let’s vote on this like a Bylaws 
change.  “Lazy majority of PMC members” applies – same as a release.

Regarding the process at 
https://cwiki.apache.org/confluence/display/METRON/Release+Process :

1. Add a step to tag the final release, as 
"apache-metron--release".

2. The current policy says that when a critical release is urgently needed, 
"the 72 hour waiting periods in Steps 7 and 8 can be waived." The formerly 
referenced Step 8 was for the Incubator vote, so that can be removed as an 
editorial issue, but we should also allow for not waiting for mirror 
propagation – let the mirrors catch up as fast as they can. So the text should 
now read: "the 72 hour waiting period in Step 7 and the wait for mirror 
propagation in Step 10 can be waived."

3. Finally, it is good practice to increment the build version in POMs 
immediately AFTER a release, so that builds with new stuff cannot be mistaken 
for builds of the release version. The current policy says to increment it just 
BEFORE a release. I suggest changing this to say:
a) immediately after a release, increment the MINOR version number (eg, 
with the 0.4.0 just released, set the new version number to 0.4.1)
b) immediately before a release, decide whether it will be a minor or major 
release. If minor, assure that the minor version number was already incremented 
after the last release and continue to use that number. If major, change the 
version number to the desired new major version. 
c) These version number changes are in master branch.  Creation of new 
branches does not occur until the idea of creating a maintenance branch or a 
new release branch has been consented by the community.

Please share your thoughts and/or vote.
Thanks,
--Matt

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[jjmeyer0]

Github user jjmeyer0 commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126841963
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java
 ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+  final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+  @Test(expected = IllegalArgumentException.class)
+  public void nullArgumentListShouldThrowException() throws Exception {
+hash.apply(null);
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void emptyArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.emptyList());
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void singleArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.singletonList("some value."));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException3() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException4() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3", "4"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void invalidAlgorithmArgumentShouldThrowException() throws 
Exception {
+hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+  }
+
+  @Test
+  public void invalidNullAlgorithmArgumentShouldThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList("value to hash", null)));
+  }
+
+  @Test
+  public void nullInputForValueToHashShouldProperlyThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList(null, "md5")));
+  }
--- End diff --

@mattf-horton just to clarify. The tests 
`invalidNullAlgorithmArgumentShouldThrowException` and 
`nullInputForValueToHashShouldProperlyThrowException` do not through 
exceptions. They return null. I originally had them throw exceptions, but 
changed it to return null, but forgot to change the names. I'll address this.

That being said, are you saying you to return a hex encoded string of 
"\u" when `toHash` is null?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #646: METRON-1030: Metron JDBC platform REST option should acce...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/646
  
I don't think we need any major changes here.  Just something more 
descriptive.

> Keep it like it is now, provide a convenient way to initialize MySQL with 
other databases requiring manual setup. In this case metron_jdbc_platform would 
accept h2, mysql, or empty meaning the user is responsible for setup.

Yes, something along these lines (^^^)  is what I think we need.  

More specifically, I would suggest...
* Change the name of `metron_jdbc_platform` to something more descriptive.  
Maybe `metron_rest_db_setup`, `metron_rest_ddl`, or something better?  

* Update the brief description in Ambari so that it reflects what you 
described above.

* Instead of an '' empty option let's call it something along the lines of 
'Manual Database Setup'.

*  Updates to the README as you already suggested.






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[jjmeyer0]

Github user jjmeyer0 commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126838847
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java
 ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+  final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+  @Test(expected = IllegalArgumentException.class)
+  public void nullArgumentListShouldThrowException() throws Exception {
+hash.apply(null);
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void emptyArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.emptyList());
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void singleArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.singletonList("some value."));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException3() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException4() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3", "4"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void invalidAlgorithmArgumentShouldThrowException() throws 
Exception {
+hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+  }
+
+  @Test
+  public void invalidNullAlgorithmArgumentShouldThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList("value to hash", null)));
+  }
+
+  @Test
+  public void nullInputForValueToHashShouldProperlyThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList(null, "md5")));
+  }
+
+  @Test
+  public void allAlgorithmsForMessageDigestShouldBeAbleToHash() throws 
Exception {
+final String valueToHash = "My value to hash";
+final Set algorithms = Security.getAlgorithms("MessageDigest");
+
+algorithms.forEach(algorithm -> {
+  try {
+final MessageDigest m = MessageDigest.getInstance(algorithm);
+m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+assertEquals(BaseEncoding.base16().encode(m.digest()), 
hash.apply(Arrays.asList(valueToHash, algorithm)));
+  } catch (NoSuchAlgorithmException e) {
+throw new RuntimeException(e);
--- End diff --

@mattf-horton thanks for the clarification. I never took the time to read 
more into it. You're comment made me take the time. thanks again!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[jjmeyer0]

Github user jjmeyer0 commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126838736
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/HashFunctionsTest.java
 ---
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang.SerializationUtils;
+import org.junit.Test;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.Security;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class HashFunctionsTest {
+  final HashFunctions.Hash hash = new HashFunctions.Hash();
+
+  @Test(expected = IllegalArgumentException.class)
+  public void nullArgumentListShouldThrowException() throws Exception {
+hash.apply(null);
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void emptyArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.emptyList());
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void singleArgumentListShouldThrowException() throws Exception {
+hash.apply(Collections.singletonList("some value."));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException3() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void argumentListWithMoreThanTwoValuesShouldThrowException4() 
throws Exception {
+hash.apply(Arrays.asList("1", "2", "3", "4"));
+  }
+
+  @Test(expected = IllegalArgumentException.class)
+  public void invalidAlgorithmArgumentShouldThrowException() throws 
Exception {
+hash.apply(Arrays.asList("value to hash", "invalidAlgorithm"));
+  }
+
+  @Test
+  public void invalidNullAlgorithmArgumentShouldThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList("value to hash", null)));
+  }
+
+  @Test
+  public void nullInputForValueToHashShouldProperlyThrowException() throws 
Exception {
+assertNull(hash.apply(Arrays.asList(null, "md5")));
+  }
+
+  @Test
+  public void allAlgorithmsForMessageDigestShouldBeAbleToHash() throws 
Exception {
+final String valueToHash = "My value to hash";
+final Set algorithms = Security.getAlgorithms("MessageDigest");
+
+algorithms.forEach(algorithm -> {
+  try {
+final MessageDigest m = MessageDigest.getInstance(algorithm);
+m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+assertEquals(BaseEncoding.base16().encode(m.digest()), 
hash.apply(Arrays.asList(valueToHash, algorithm)));
+  } catch (NoSuchAlgorithmException e) {
+throw new RuntimeException(e);
+  }
+});
+  }
+
+  @Test
+  public void 
allAlgorithmsForMessageDigestShouldBeAbleToHashDirectStellarCall() throws 
Exception {
+final String valueToHash = "My value to hash";
+final Set algorithms = Security.getAlgorithms("MessageDigest");
+
+algorithms.forEach(algorithm -> {
+  try {
+final Object actual = run("HASH('" + valueToHash + "', '" + 
algorithm + "')", Collections.emptyMap());
+
+final MessageDigest m = MessageDigest.getInstance(algorithm);
+m.update(valueToHash.getBytes(StandardCharsets.UTF_8));
+
+

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126836524
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+  @Stellar(
+name = "HASH",
+description = "Hashes a given value using the given hashing algorithm 
and returns a hex encoded string. This function only hashes " +
+  "strings and values that implement java.io.Serializable.",
+params = {
+  "toHash - value to hash.",
+  "hashType - A valid string representation of an algorithm supported 
by java.security.MessageDigest.",
+},
+returns = "A hex representation of a hashed value using the given 
hashing algorithm. If either argument is " +
+  "null then null will be returned. If the type of 'toHash' is neither 
a string nor of type java.io.Serializable, " +
+  "then null is returned."
+  )
+  public static class Hash extends BaseStellarFunction {
+
+@Override
+public Object apply(final List args) {
+  if (args == null || args.size() != 2) {
+throw new IllegalArgumentException("Invalid number of arguments: " 
+ (args == null ? 0 : args.size()));
+  }
+
+  final Object toHash = args.get(0);
+  final Object hashType = args.get(1);
+
+  if (toHash == null || hashType == null) {
+return null;
+  }
+
+  final MessageDigest messageDigest = 
getMessageDigest(hashType.toString());
+
+  if (toHash instanceof String) {
+return getHashAsHex(messageDigest, 
toHash.toString().getBytes(StandardCharsets.UTF_8));
+  } else if (toHash instanceof Serializable) {
+final byte[] serialized = 
SerializationUtils.serialize((Serializable) toHash);
+return getHashAsHex(messageDigest, serialized);
+  }
+
+  return null;
+}
+
+private MessageDigest getMessageDigest(String hashType) {
--- End diff --

if you were on irc we wound't have these problems ;)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126836440
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+  @Stellar(
+name = "HASH",
+description = "Hashes a given value using the given hashing algorithm 
and returns a hex encoded string. This function only hashes " +
+  "strings and values that implement java.io.Serializable.",
+params = {
+  "toHash - value to hash.",
+  "hashType - A valid string representation of an algorithm supported 
by java.security.MessageDigest.",
+},
+returns = "A hex representation of a hashed value using the given 
hashing algorithm. If either argument is " +
+  "null then null will be returned. If the type of 'toHash' is neither 
a string nor of type java.io.Serializable, " +
+  "then null is returned."
+  )
+  public static class Hash extends BaseStellarFunction {
+
+@Override
+public Object apply(final List args) {
+  if (args == null || args.size() != 2) {
+throw new IllegalArgumentException("Invalid number of arguments: " 
+ (args == null ? 0 : args.size()));
+  }
+
+  final Object toHash = args.get(0);
+  final Object hashType = args.get(1);
+
+  if (toHash == null || hashType == null) {
+return null;
+  }
+
+  final MessageDigest messageDigest = 
getMessageDigest(hashType.toString());
+
+  if (toHash instanceof String) {
+return getHashAsHex(messageDigest, 
toHash.toString().getBytes(StandardCharsets.UTF_8));
+  } else if (toHash instanceof Serializable) {
+final byte[] serialized = 
SerializationUtils.serialize((Serializable) toHash);
+return getHashAsHex(messageDigest, serialized);
+  }
+
+  return null;
+}
+
+private MessageDigest getMessageDigest(String hashType) {
--- End diff --

I have this 
[Encodings.java](https://github.com/apache/metron/pull/642/files#diff-21626fcded1721297dd93114ccfd5ba1)
  and 
[EncodingFunctions.java](https://github.com/apache/metron/pull/642/files#diff-295147ddc21d558ddd7af7f31d9bb8ca)
 fresh in my mind.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126835535
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+  @Stellar(
+name = "HASH",
+description = "Hashes a given value using the given hashing algorithm 
and returns a hex encoded string. This function only hashes " +
+  "strings and values that implement java.io.Serializable.",
+params = {
+  "toHash - value to hash.",
+  "hashType - A valid string representation of an algorithm supported 
by java.security.MessageDigest.",
+},
+returns = "A hex representation of a hashed value using the given 
hashing algorithm. If either argument is " +
+  "null then null will be returned. If the type of 'toHash' is neither 
a string nor of type java.io.Serializable, " +
+  "then null is returned."
+  )
+  public static class Hash extends BaseStellarFunction {
+
+@Override
+public Object apply(final List args) {
+  if (args == null || args.size() != 2) {
+throw new IllegalArgumentException("Invalid number of arguments: " 
+ (args == null ? 0 : args.size()));
+  }
+
+  final Object toHash = args.get(0);
+  final Object hashType = args.get(1);
+
+  if (toHash == null || hashType == null) {
+return null;
+  }
+
+  final MessageDigest messageDigest = 
getMessageDigest(hashType.toString());
+
+  if (toHash instanceof String) {
+return getHashAsHex(messageDigest, 
toHash.toString().getBytes(StandardCharsets.UTF_8));
+  } else if (toHash instanceof Serializable) {
+final byte[] serialized = 
SerializationUtils.serialize((Serializable) toHash);
+return getHashAsHex(messageDigest, serialized);
+  }
+
+  return null;
+}
+
+private MessageDigest getMessageDigest(String hashType) {
--- End diff --

Never mind, I think I misunderstood you again. You were probably just 
proposing the code be in a separate directory instead of in the generic 
stellar/dsl/functions, right?  If so then I'm in left field here.  Prob didn't 
get enough sleep last night...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126834205
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+  @Stellar(
+name = "HASH",
--- End diff --

Ah, sorry I didn't understand this was only a comment about documentation.  
I guess you're right; since most objects are indeed serializable, maybe it's 
okay to let them discover the occasional object that isn't (if any).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #646: METRON-1030: Metron JDBC platform REST option should acce...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/646
  
Hive provides a list of supported databases:  
https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-Local/EmbeddedMetastoreDatabase(Derby)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #646: METRON-1030: Metron JDBC platform REST option should acce...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/646
  
Your fix, plus some clarity in the documentation / ambari label etc + some 
idea that we need to think of something long term would be good right?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126828810
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+  @Stellar(
+name = "HASH",
--- End diff --

Still, if it's not serializable, what should HASH do?  This may be moot, 
since all normal objects Stellar can get its hands on are, I think, 
Serializable.  The exceptions may indeed be appropriate to not be able to hash.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #646: METRON-1030: Metron JDBC platform REST option should acce...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/646
  
What do other HDP / ambari configured services do to support multiple 
databases? There are other services/projects we can look to for ideas



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #646: METRON-1030: Metron JDBC platform REST option should acce...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/646
  
this is a general problem I think


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #646: METRON-1030: Metron JDBC platform REST option should acce...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/646
  
It needs to be documented in the ambari ui, if it needs clarification.  The 
user in ambari isn't going to be using the readme


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #646: METRON-1030: Metron JDBC platform REST option should acce...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/646
  
@ottobackwards it's in metron-rest README but I can expand it a little bit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126810315
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+  @Stellar(
+name = "HASH",
+description = "Hashes a given value using the given hashing algorithm 
and returns a hex encoded string. This function only hashes " +
+  "strings and values that implement java.io.Serializable.",
+params = {
+  "toHash - value to hash.",
+  "hashType - A valid string representation of an algorithm supported 
by java.security.MessageDigest.",
+},
--- End diff --

may want to call out that it is a string of the hex value


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126810010
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+  @Stellar(
+name = "HASH",
--- End diff --

I am not sure that java.io.Serializable should bleed through here.  While 
we may limit ourselves in some places to having stellar conform to java 
limitations and behaviors, and other things, we don't really limit ourselves 
explicitly to java types.  The idea that the user 'knows' the type of the 
entity in a stellar statement in java doesn't seem right.

While this limitation may be true and enforced, I am not sure it should be 
expressed this way explicitly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #646: METRON-1030: Metron JDBC platform REST option should acce...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/646
  
The comment "This is needed when configuring MySQL for Metron REST because 
this property must be" is actually incorrect now that I go back and look at it. 
 The schema-mysql.sql was initially incorrect because it was dropping the 
tables (MySQL doesn't support "create table if not exists") and recreating them 
every time.  

Fortunately Spring just ignores any errors thrown when running these 
scripts so the drop statements were just removed and creating a table that 
already existed is safely ignored.  This is just a convenience feature that 
keeps people from having to manually create database assets.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #646: METRON-1030: Metron JDBC platform REST option should acce...

[lvets]

Github user lvets commented on the issue:

https://github.com/apache/metron/pull/646
  
"This is needed when configuring MySQL for Metron REST because this 
property must be '' and not 'mysql'.". So this needs to be empty when you use 
mysql? I'm not following... :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #646: METRON-1030: Metron JDBC platform REST option should acce...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/646
  
That seems like a configuration that is rather confusing... 
Is there a way we can clarify that?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #646: METRON-1030: Metron JDBC platform REST option should acce...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/646
  
Spring provides a mechanism for running sql scripts for different databases 
on startup.  This is useful for initializing tables and such.  We provide 2 
right now:  schema-h2.sql and schema-mysql.sql.  If you set this property to 
"h2" it will run the schema-h2.sql script automatically at startup.  Same goes 
for mysql.  It's possible that someone may want to use a different database and 
in that case they would want to set this to a blank string.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #646: METRON-1030: Metron JDBC platform REST option should acce...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/646
  
^^^ 
" This is needed when configuring MySQL for Metron REST because this 
property must be '' and not 'mysql'."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #646: METRON-1030: Metron JDBC platform REST option should acce...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/646
  
When would a user want to set this property to an empty string?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #634: METRON-1017: Ambari components should be separate

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/634
  
+1 Looks sharp!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #634: METRON-1017: Ambari components should be separate

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/634#discussion_r126802506
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-enrichment-env.xml
 ---
@@ -0,0 +1,174 @@
+
+
+
+
+  
+geoip_url
+
http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
+Location of the GeoIP data to load.
+GEOIP Load Datafile URL
+  
+  
+enrichment_host_known_hosts
+List of Known Hosts for Host Enrichment
+[{"ip":"10.1.128.236", "local":"YES", "type":"webserver", 
"asset_value" : "important"},{"ip":"10.1.128.237", "local":"UNKNOWN", 
"type":"unknown", "asset_value" : "important"},{"ip":"10.60.10.254", 
"local":"YES", "type":"printer", "asset_value" : "important"}]
+Host Enrichment
+
+  content
+
+  
+  
+enrichment_kafka_start
+Enrichment Topology Spout Offset
+UNCOMMITTED_EARLIEST
+Enrichment Offset
--- End diff --

Not a big deal.  I think we can address it later if needed.  Thanks for 
checking.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #634: METRON-1017: Ambari components should be separate

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/634#discussion_r126802519
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
 ---
@@ -77,132 +71,17 @@
 Metron pid dir
 
 
-metron_rest_port
-8082
-Port the REST application runs on
-Metron REST port
-
-
-metron_management_ui_port
-4200
-Port the Management UI runs on
-Management UI port
-
-
-metron_jvm_flags
-JVM flags passed to Metron scripts
-Metron JVM flags
-
-
-true
-
-
-
-metron_spring_profiles_active
-Active Spring profiles
-Active Spring profiles
-
-
-true
-
-
-
-metron_jdbc_driver
-
-Class name of the JDBC Driver used by 
Metron
-Metron JDBC Driver
-
-
-metron_jdbc_url
-
-JDBC Connection URL used by Metron
-Metron JDBC URL
-
-
-metron_jdbc_username
-
-Metron JDBC Username
-Metron JDBC username
-
-
-metron_jdbc_password
-
-PASSWORD
-Metron JDBC Password
-Metron JDBC password
-
-password
-false
-METRON_CLIENT
-
-
-
-
-metron_jdbc_platform
-
-Database platform used by Metron.  One of: hsqldb, 
h2, oracle, mysql, postgresql
-Metron JDBC platform
-
-
-metron_jdbc_client_path
-Path to JDBC jar for selected platform
-Metron JDBC client path
-
-
-true
-
-
-
-metron_temp_grok_path
-Temporary local file path where grok patterns are 
written during testing
-./patterns/temp
-Metron temp grok path
-
-
-metron_default_grok_path
-Default HDFS directory path used when storing Grok 
patterns
-/apps/metron/patterns
-Metron default grok path
-
-
-metron_spring_options
-Additional Spring options not included in the 
rest_application.yml file
-Metron Spring options
-
-
-true
-
-
-
 metron_topic_retention
 Kafka Retention in GB
 10
 Topic Retention
 
 
-parsers
-bro,snort,yaf
-Metron parsers to deploy
-Metron Parsers
-
-
-metron_indexing_topology
-indexing
-The Storm topology name for Indexing
-Indexing Topology Name
-
-
 es_cluster_name
 metron
 Name of Elasticsearch Cluster
 Elasticsearch Cluster Name
 
-
-geoip_url
-
http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
-Location of the GeoIP data to load.
-GEOIP Load Datafile URL
-
 
 es_hosts
--- End diff --

I think that is just fine for now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #636: METRON-1022: Elasticsearch REST endpoint

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/636
  
The latest commit enforces a maximum page size.  A value of 1000 is set by 
default.  As it currently stands, you would need to add this property to the 
rest_application.yml file manually if it needed to be changed.  I will expose 
it as a property in Ambari as soon as METRON-1017 makes it in.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #634: METRON-1017: Ambari components should be separate

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/634#discussion_r126789624
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-enrichment-env.xml
 ---
@@ -0,0 +1,174 @@
+
+
+
+
+  
+geoip_url
+
http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
+Location of the GeoIP data to load.
+GEOIP Load Datafile URL
+  
+  
+enrichment_host_known_hosts
+List of Known Hosts for Host Enrichment
+[{"ip":"10.1.128.236", "local":"YES", "type":"webserver", 
"asset_value" : "important"},{"ip":"10.1.128.237", "local":"UNKNOWN", 
"type":"unknown", "asset_value" : "important"},{"ip":"10.60.10.254", 
"local":"YES", "type":"printer", "asset_value" : "important"}]
+Host Enrichment
+
+  content
+
+  
+  
+enrichment_kafka_start
+Enrichment Topology Spout Offset
+UNCOMMITTED_EARLIEST
+Enrichment Offset
--- End diff --

I looked at this and the options are readable if you click the dropdown.  
Would be better if the whole value was displayed but I'm not even sure how to 
make the dropdown wider.  I can look into it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #634: METRON-1017: Ambari components should be separate

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/634#discussion_r126788288
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py
 ---
@@ -94,24 +96,24 @@ def init_hdfs_dir(self):
 Logger.info('Done creating HDFS indexing directory')
 
 def start_indexing_topology(self):
-Logger.info("Starting Metron indexing topology: 
{0}".format(self.__indexing))
+Logger.info("Starting Metron indexing topology: 
{0}".format(self.__indexing_topology))
 start_cmd_template = """{0}/bin/start_elasticsearch_topology.sh \
 -s {1} \
 -z {2}"""
-Logger.info('Starting ' + self.__indexing)
+Logger.info('Starting ' + self.__indexing_topology)
 if self.__params.security_enabled:
 metron_security.kinit(self.__params.kinit_path_local,
   self.__params.metron_keytab_path,
   self.__params.metron_principal_name,
   execute_user=self.__params.metron_user)
-Execute(start_cmd_template.format(self.__params.metron_home, 
self.__indexing, self.__params.zookeeper_quorum),
+Execute(start_cmd_template.format(self.__params.metron_home, 
self.__indexing_topology, self.__params.zookeeper_quorum),
 user=self.__params.metron_user)
 
 Logger.info('Finished starting indexing topology')
 
 def stop_indexing_topology(self):
-Logger.info('Stopping ' + self.__indexing)
-stop_cmd = 'storm kill ' + self.__indexing
+Logger.info('Stopping ' + self.__indexing_topology)
--- End diff --

Yes exactly.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #638: METRON-933 New stellar function for regex group ca...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/638


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Metron REST - Logging Config

[Nick Allen]

How do I configure logging for Metron REST on a deployed host?

Right now a log4j.properties file gets packaged into the metron-rest JAR
itself.  Is there is an easy way that I am missing?

[GitHub] metron pull request #638: METRON-933 New stellar function for regex group ca...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/638#discussion_r126778701
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/RegExFunctionsTest.java
 ---
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import org.apache.metron.stellar.dsl.ParseException;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.runPredicate;
+
+public class RegExFunctionsTest {
+
+  // test RegExMatch
+  @Test
+  public void testRegExMatch() throws Exception {
+final Map variableMap = new HashMap() 
{{
+  put("numbers", "12345");
+  put("numberPattern", "\\d(\\d)(\\d).*");
--- End diff --

Ahh, right sorry.  So no need to do this when specifying within Stellar, 
just in the tests - sounds good.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #638: METRON-933 New stellar function for regex group capture

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/638
  
+1 via inspection


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #638: METRON-933 New stellar function for regex group ca...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/638#discussion_r126778367
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RegExFunctions.java
 ---
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import org.apache.metron.stellar.common.utils.ConversionUtils;
+import org.apache.metron.stellar.common.utils.PatternCache;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+public class RegExFunctions {
+
+  @Stellar(name = "REGEXP_MATCH",
+  description = "Determines whether a regex matches a string",
+  params = {
+  "string - The string to test",
+  "pattern - The proposed regex pattern"
+  },
+  returns = "True if the regex pattern matches the string and false if 
otherwise.")
+  public static class RegexpMatch extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+  if (list.size() < 2) {
+throw new IllegalStateException(
+"REGEXP_MATCH expects two args: [string, pattern] where 
pattern is a regexp pattern");
+  }
+  String patternString = (String) list.get(1);
+  String str = (String) list.get(0);
+  if (str == null || patternString == null) {
+return false;
+  }
+  return 
PatternCache.INSTANCE.getPattern(patternString).matcher(str).matches();
+}
+  }
+
+  @Stellar(name = "REGEXP_GROUP_VAL",
+  description = "Returns the value of a group in a regex against a 
string",
+  params = {
+  "string - The string to test",
+  "pattern - The proposed regex pattern",
+  "group - integer that selects what group to select, starting at 
1"
+  },
+  returns = "The value of the group, or null if not matched or no 
group at index")
+  public static class RegexpGroupValue extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+  if (list.size() < 3) {
--- End diff --

Addressed, good catch


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #638: METRON-933 New stellar function for regex group ca...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/638#discussion_r126778032
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/RegExFunctionsTest.java
 ---
@@ -0,0 +1,70 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import org.apache.metron.stellar.dsl.ParseException;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.runPredicate;
+
+public class RegExFunctionsTest {
+
+  // test RegExMatch
+  @Test
+  public void testRegExMatch() throws Exception {
+final Map variableMap = new HashMap() 
{{
+  put("numbers", "12345");
+  put("numberPattern", "\\d(\\d)(\\d).*");
--- End diff --

In java I need to \\ to escape
http://docs.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Upgrade from 0.4.0-rc to 0.4.0-release

[Laurens Vets]

Can someone confirm that the following instructions are correct for 
upgrading from 0.4.0-rc to 0.4.0-release? They seem to work for me, but 
my testing is rather limited.


# cd metron
# git checkout Metron_0.4.0
# mvn clean package -DskipTests -T 2C -P HDP-2.5.0.0,mpack
# cd metron-deployment/packaging/docker/rpm-docker
# mvn clean install -DskipTests -PHDP-2.5.0.0

# mkdir /localrepo/archive
# mv /localrepo/*.rpm /localrepo/archive
# cp -rp 
/root/metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/* 
/localrepo/

# cd /localrepo
# createrepo /localrepo

# ambari-server stop
# ambari-server install-mpack 
--mpack=/root/metron/metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-0.4.0.0.tar.gz 
--verbose --force


Old 0.4.0-rc RPMs:
[root@metron1 localrepo]#  rpm -qa | grep metron
metron-data-management-0.4.0-201707052247.noarch
metron-profiler-0.4.0-201707052247.noarch
metron-indexing-0.4.0-201707052247.noarch
metron-parsers-0.4.0-201707052247.noarch
metron-elasticsearch-0.4.0-201707052247.noarch
metron-enrichment-0.4.0-201707052247.noarch
metron-rest-0.4.0-201707052247.noarch
metron-config-0.4.0-201707052247.noarch
metron-solr-0.4.0-201707052247.noarch
metron-pcap-0.4.0-201707052247.noarch
metron-common-0.4.0-201707052247.noarch
[root@metron1 localrepo]#

# rpm -Uvh metron*.rpm

New 0.4.0-release RPMs installed:
[root@metron1 localrepo]#  rpm -qa | grep metron
metron-config-0.4.0-201707111641.noarch
metron-pcap-0.4.0-201707111641.noarch
metron-enrichment-0.4.0-201707111641.noarch
metron-indexing-0.4.0-201707111641.noarch
metron-solr-0.4.0-201707111641.noarch
metron-data-management-0.4.0-201707111641.noarch
metron-elasticsearch-0.4.0-201707111641.noarch
metron-profiler-0.4.0-201707111641.noarch
metron-common-0.4.0-201707111641.noarch
metron-rest-0.4.0-201707111641.noarch
metron-parsers-0.4.0-201707111641.noarch
You have new mail in /var/spool/mail/root
[root@metron1 localrepo]#

# ambari-server start

[GitHub] metron pull request #646: METRON-1030: Metron JDBC platform REST option shou...

[merrimanr]

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/646

METRON-1030: Metron JDBC platform REST option should accept a blank value

## Contributor Comments
Ambari should allow the metron_jdbc_platform option to be set to '' which 
it currently does not.  To verify, you should be able to spin up full dev, 
navigate to the REST tab and set that property to ''.  This is needed when 
configuring MySQL for Metron REST because this property must be '' and not 
'mysql'.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-1030

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/646.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #646


commit 2247268be85bc809180a86eb661cdc7e116f14b3
Author: merrimanr 
Date:   2017-07-11T17:26:03Z

added empty-value-valid option to metron_jdbc_platform




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #638: METRON-933 New stellar function for regex group ca...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/638#discussion_r126739325
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RegExFunctions.java
 ---
@@ -0,0 +1,103 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import org.apache.metron.stellar.common.utils.ConversionUtils;
+import org.apache.metron.stellar.common.utils.PatternCache;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+public class RegExFunctions {
+
+  @Stellar(name = "REGEXP_MATCH",
+  description = "Determines whether a regex matches a string",
+  params = {
+  "string - The string to test",
+  "pattern - The proposed regex pattern"
+  },
+  returns = "True if the regex pattern matches the string and false if 
otherwise.")
+  public static class RegexpMatch extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+  if (list.size() < 2) {
+throw new IllegalStateException(
+"REGEXP_MATCH expects two args: [string, pattern] where 
pattern is a regexp pattern");
+  }
+  String patternString = (String) list.get(1);
+  String str = (String) list.get(0);
+  if (str == null || patternString == null) {
+return false;
+  }
+  return 
PatternCache.INSTANCE.getPattern(patternString).matcher(str).matches();
+}
+  }
+
+  @Stellar(name = "REGEXP_GROUP_VAL",
+  description = "Returns the value of a group in a regex against a 
string",
+  params = {
+  "string - The string to test",
+  "pattern - The proposed regex pattern",
+  "group - integer that selects what group to select, starting at 
1"
+  },
+  returns = "The value of the group, or null if not matched or no 
group at index")
+  public static class RegexpGroupValue extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+  if (list.size() < 3) {
--- End diff --

Why isn't this `!=`, since any arguments after the 3rd are ignored?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #645: METRON-1029: Topology worker opts are not being se...

[merrimanr]

GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/645

METRON-1029: Topology worker opts are not being set when Kerberos is enabled

## Contributor Comments
This PR properly sets the topology worker opts property for enrichment and 
indexing topologies.  To test, spin up full dev and enable kerberos.  Both 
enrichment and indexing should run without error.  I also added other worker 
options in Ambari and verified the topologies still functioned without error.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-1029

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/645.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #645


commit cdbbd8321d3ceb4340182a9c207e0d0925d924c8
Author: merrimanr 
Date:   2017-07-11T17:24:05Z

added client_jaas.conf option to enrichment and indexing worker opts 
property




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #621: METRON-1001: Allow metron to ingest parser metadat...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/621


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #621: METRON-1001: Allow metron to ingest parser metadata along...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/621
  
@cestella Nope, I'm good, +1.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #641: METRON-539: added HASH function for stellar.

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/641#discussion_r126752781
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/HashFunctions.java
 ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import com.google.common.io.BaseEncoding;
+import org.apache.commons.lang3.SerializationUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.List;
+
+public class HashFunctions {
+  @Stellar(
+name = "HASH",
+description = "Hashes a given value using the given hashing algorithm 
and returns a hex encoded string. This function only hashes " +
+  "strings and values that implement java.io.Serializable.",
+params = {
+  "toHash - value to hash.",
+  "hashType - A valid string representation of an algorithm supported 
by java.security.MessageDigest.",
+},
+returns = "A hex representation of a hashed value using the given 
hashing algorithm. If either argument is " +
+  "null then null will be returned. If the type of 'toHash' is neither 
a string nor of type java.io.Serializable, " +
+  "then null is returned."
+  )
+  public static class Hash extends BaseStellarFunction {
--- End diff --

Thanks @ottobackwards , I agree.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #621: METRON-1001: Allow metron to ingest parser metadata along...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/621
  
@justinleet any other comments or concerns?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #644: METRON-1027: Errant Log Message When No GeoIP Hit ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/644


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #644: METRON-1027: Errant Log Message When No GeoIP Hit on Vali...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/644
  
Also, ran this up on Full Dev and did some quick validation on `GEO_GET` 
just to be sure.

```
[Stellar]>>> GEO_GET("192.168.0.1")
2017-07-11 15:11:13,401 INFO  [AeshProcess: 3] 
stellar.GeoEnrichmentFunctions: Initializing GeoEnrichmentFunctions
2017-07-11 15:11:13,790 WARN  [AeshProcess: 3] util.NativeCodeLoader: 
Unable to load native-hadoop library for your platform... using builtin-java 
classes where applicable
2017-07-11 15:11:14,636 WARN  [AeshProcess: 3] 
shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot 
be used because libhadoop cannot be loaded.
{}
[Stellar]>>> GEO_GET("75.118.136.186")
{country=US, dmaCode=535, city=Columbus, postalCode=43230, 
latitude=40.0345, location_point=40.0345,-82.8695, locID=4509177, 
longitude=-82.8695}
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #634: METRON-1017: Ambari components should be separate

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/634#discussion_r126689168
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-enrichment-env.xml
 ---
@@ -0,0 +1,174 @@
+
+
+
+
+  
+geoip_url
+
http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
+Location of the GeoIP data to load.
+GEOIP Load Datafile URL
+  
+  
+enrichment_host_known_hosts
+List of Known Hosts for Host Enrichment
+[{"ip":"10.1.128.236", "local":"YES", "type":"webserver", 
"asset_value" : "important"},{"ip":"10.1.128.237", "local":"UNKNOWN", 
"type":"unknown", "asset_value" : "important"},{"ip":"10.60.10.254", 
"local":"YES", "type":"printer", "asset_value" : "important"}]
+Host Enrichment
+
+  content
+
+  
+  
+enrichment_kafka_start
+Enrichment Topology Spout Offset
+UNCOMMITTED_EARLIEST
+Enrichment Offset
--- End diff --

With Ambari's drop down selector, all I can read is "UNCOMMITTE".  I can't 
actually tell if it is UNCOMITTED_EARLIEST or UNCOMMITTED_LATEST.  Is that a 
pre-existing condition?




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #634: METRON-1017: Ambari components should be separate

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/634#discussion_r126687833
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/indexing_commands.py
 ---
@@ -94,24 +96,24 @@ def init_hdfs_dir(self):
 Logger.info('Done creating HDFS indexing directory')
 
 def start_indexing_topology(self):
-Logger.info("Starting Metron indexing topology: 
{0}".format(self.__indexing))
+Logger.info("Starting Metron indexing topology: 
{0}".format(self.__indexing_topology))
 start_cmd_template = """{0}/bin/start_elasticsearch_topology.sh \
 -s {1} \
 -z {2}"""
-Logger.info('Starting ' + self.__indexing)
+Logger.info('Starting ' + self.__indexing_topology)
 if self.__params.security_enabled:
 metron_security.kinit(self.__params.kinit_path_local,
   self.__params.metron_keytab_path,
   self.__params.metron_principal_name,
   execute_user=self.__params.metron_user)
-Execute(start_cmd_template.format(self.__params.metron_home, 
self.__indexing, self.__params.zookeeper_quorum),
+Execute(start_cmd_template.format(self.__params.metron_home, 
self.__indexing_topology, self.__params.zookeeper_quorum),
 user=self.__params.metron_user)
 
 Logger.info('Finished starting indexing topology')
 
 def stop_indexing_topology(self):
-Logger.info('Stopping ' + self.__indexing)
-stop_cmd = 'storm kill ' + self.__indexing
+Logger.info('Stopping ' + self.__indexing_topology)
--- End diff --

I just want to make sure I understand the changes you made here.  Seems 
like you fixed a bug, but just want to be sure.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #634: METRON-1017: Ambari components should be separate

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/634#discussion_r126687126
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
 ---
@@ -77,132 +71,17 @@
 Metron pid dir
 
 
-metron_rest_port
-8082
-Port the REST application runs on
-Metron REST port
-
-
-metron_management_ui_port
-4200
-Port the Management UI runs on
-Management UI port
-
-
-metron_jvm_flags
-JVM flags passed to Metron scripts
-Metron JVM flags
-
-
-true
-
-
-
-metron_spring_profiles_active
-Active Spring profiles
-Active Spring profiles
-
-
-true
-
-
-
-metron_jdbc_driver
-
-Class name of the JDBC Driver used by 
Metron
-Metron JDBC Driver
-
-
-metron_jdbc_url
-
-JDBC Connection URL used by Metron
-Metron JDBC URL
-
-
-metron_jdbc_username
-
-Metron JDBC Username
-Metron JDBC username
-
-
-metron_jdbc_password
-
-PASSWORD
-Metron JDBC Password
-Metron JDBC password
-
-password
-false
-METRON_CLIENT
-
-
-
-
-metron_jdbc_platform
-
-Database platform used by Metron.  One of: hsqldb, 
h2, oracle, mysql, postgresql
-Metron JDBC platform
-
-
-metron_jdbc_client_path
-Path to JDBC jar for selected platform
-Metron JDBC client path
-
-
-true
-
-
-
-metron_temp_grok_path
-Temporary local file path where grok patterns are 
written during testing
-./patterns/temp
-Metron temp grok path
-
-
-metron_default_grok_path
-Default HDFS directory path used when storing Grok 
patterns
-/apps/metron/patterns
-Metron default grok path
-
-
-metron_spring_options
-Additional Spring options not included in the 
rest_application.yml file
-Metron Spring options
-
-
-true
-
-
-
 metron_topic_retention
 Kafka Retention in GB
 10
 Topic Retention
 
 
-parsers
-bro,snort,yaf
-Metron parsers to deploy
-Metron Parsers
-
-
-metron_indexing_topology
-indexing
-The Storm topology name for Indexing
-Indexing Topology Name
-
-
 es_cluster_name
 metron
 Name of Elasticsearch Cluster
 Elasticsearch Cluster Name
 
-
-geoip_url
-
http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
-Location of the GeoIP data to load.
-GEOIP Load Datafile URL
-
 
 es_hosts
--- End diff --

When I changed the 'Elasticsearch Hosts' under 'Index Settings', Ambari 
wanted to restart all Metron services.  Should it just want to restart Indexing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r126672638
  
--- Diff: metron-interface/metron-alerts/src/app/model/alert.ts ---
@@ -0,0 +1,44 @@
+export class Alert {
--- End diff --

I haven't been a part of the review a whole lot, so forgive me if I'm 
asking a question that's already been answered.  This `Alert` class seems very 
Snort specific, but `is_alert` can potentially show up as a result of any of 
our input data.  Shouldn't this be relatively abstract and limited to the 
fields that are essential for the alert?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

[mraliagha]

Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/620
  
@iraghumitra I've tested your latest commit and it is much better now. 
However, I cannot see all the fields in the customise visible fields panel.


![image](https://user-images.githubusercontent.com/8438293/28059526-dcb212a2-6667-11e7-94b1-9e65eee43be5.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---