date:20170713

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

2017-07-13 Thread mraliagha

Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/620
  
@iraghumitra You're welcome.

As a load balancer. For Elasticsearch client, if you provide a list of 
endpoints, it acts as a load balancer to make sure one of them will not get 
overwhelmed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #623: METRON-1003 ParserUtil parses dates incorrect

2017-07-13 Thread bjigmp

Github user bjigmp commented on a diff in the pull request:

https://github.com/apache/metron/pull/623#discussion_r127359171
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/utils/ParserUtils.java
 ---
@@ -55,7 +55,7 @@ public static Long convertToEpoch(String m, String d, 
String ts,
 Date date = new SimpleDateFormat("MMM", Locale.ENGLISH).parse(m);
 Calendar cal = Calendar.getInstance();
 cal.setTime(date);
-String month = String.valueOf(cal.get(Calendar.MONTH));
+String month = String.valueOf(cal.get(Calendar.MONTH) + 1);
--- End diff --

Done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #655: Profiler Should be Less Dramatic When Missing Conf...

2017-07-13 Thread nickwallen

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/655

Profiler Should be Less Dramatic When Missing Configuration

When the Profiler topology is running, but no Profiler definition is found 
in Zookeeper, the Profiler will now just calmly log a warning.  Previously the 
Profiler would throw a rather nasty looking exception.

I also refactored the logic for Profile `expires` handling.  Previously, 
this code was not all in one place (`ProfileHBaseMapper` + `ProfileConfig`), 
which made it less than clear what units the code base supported for `expires`. 
  I refactored the code so that it is all in one place (`ProfileHBaseMapper`) 
and very clear what is happening to avoid confusion.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1043

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/655.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #655


commit 8d77e4167aeefc1a769c1c266ff8e883e9a06ce8
Author: Nick Allen 
Date:   2017-07-13T19:45:16Z

Profiler Should be Less Dramatic When Missing Configuration




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #642: METRON-984 Stellar functions to decode encoded fields or ...

2017-07-13 Thread ottobackwards

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/642
  
Ok @cestella , I used the Stellar management functions : 
```bash
[Stellar]>>> bro_parser_config:= CONFIG_GET('PARSER','bro')
[Stellar]>>> PARSER_STELLAR_TRANSFORM_PRINT( bro_parser_config )

âââââââââ¤âââââââââââââââââ
â Field â Transformation â
â 
ââââââââ§âââââââââââââââââ£
â (empty)â

ââââââââââââââââââââââââââ
[Stellar]>>> original_string:='foo'
[Stellar]>>> encoded_original:=ENCODE(original_string,'BASE64')
[Stellar]>>> SHELL_LIST_VARS(40)

âââââââââââââââââââââ¤ââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ¤âââââââââââââââââââââââââââââââââââ
â VARIABLE  â VALUE 
â EXPRESSION   â
â 
ââââââââââââââââââââªââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââªâââââââââââââââââââââââââââââââââââ£
â encoded_original  â Zm9v  
â ENCODE(original_string,'BASE64') â

âââââââââââââââââââââ¼ââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ¼âââââââââââââââââââââââââââââââââââ¢
â original_string   â foo   
â 'foo'â

âââââââââââââââââââââ¼ââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ¼âââââââââââââââââââââââââââââââââââ¢
â bro_parser_config â { 
â CONFIG_GET('PARSER','bro')   â
â   â   
â  â
â   â 
"parserClassName":"org.apache.metron.parsers.bro.BasicBroParser", â   
   â
â   â   
â  â
â   â "sensorTopic":"bro",  
â  â
â   â   "parserConfig": 
â  â
â   â {}
â  â
â   â } 
â  â

âââââââââââââââââââââ§ââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââââ§âââââââââââââââââââââââââââââââââââ
[Stellar]>>> bro_parser_config_new:= PARSER_STELLAR_TRANSFORM_ADD( 
bro_parser_config , SHELL_VARS2MAP( 'encoded_original' ) )
[Stellar]>>> PARSER_STELLAR_TRANSFORM_PRINT( bro_parser_config_new )

ââââââââââââââââââââ¤âââââââââââââââââââââââââââââââââââ
â Fieldâ Transformation   â
â 
âââââââââââââââââââªâââââââââââââââââââââââââââââââââââ£
â encoded_original â ENCODE(original_string,'BASE64') â

[GitHub] metron issue #642: METRON-984 Stellar functions to decode encoded fields or ...

2017-07-13 Thread ottobackwards

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/642
  
And they do get returned from the rest api stellar controller



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

2017-07-13 Thread nickwallen

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127331446
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/RowKeyBuilderFactory.java
 ---
@@ -0,0 +1,125 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.beanutils.PropertyUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.InvocationTargetException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_ROW_KEY_BUILDER;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * A Factory class that can create a RowKeyBuilder based on global 
property values.
+ */
+public class RowKeyBuilderFactory {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(RowKeyBuilderFactory.class);
+
+  /**
+   * Create a RowKeyBuilder.
+   * @param global The global properties.
+   * @return A RowKeyBuilder instantiated using the global property values.
+   */
+  public static RowKeyBuilder create(Map global) {
+String rowKeyBuilderClass = PROFILER_ROW_KEY_BUILDER.get(global, 
String.class);
+LOG.debug("profiler client: {}={}", PROFILER_ROW_KEY_BUILDER, 
rowKeyBuilderClass);
+
+// instantiate the RowKeyBuilder
+RowKeyBuilder builder = 
ReflectionUtils.createInstance(rowKeyBuilderClass);
+setSaltDivisor(global, builder);
+setPeriodDuration(global, builder);
--- End diff --

If I had some IoC-like functionality like Flux or Spring here, then this 
wouldn't be a problem at all.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

2017-07-13 Thread nickwallen

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127326080
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/RowKeyBuilderFactory.java
 ---
@@ -0,0 +1,125 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.beanutils.PropertyUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.InvocationTargetException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_ROW_KEY_BUILDER;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * A Factory class that can create a RowKeyBuilder based on global 
property values.
+ */
+public class RowKeyBuilderFactory {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(RowKeyBuilderFactory.class);
+
+  /**
+   * Create a RowKeyBuilder.
+   * @param global The global properties.
+   * @return A RowKeyBuilder instantiated using the global property values.
+   */
+  public static RowKeyBuilder create(Map global) {
+String rowKeyBuilderClass = PROFILER_ROW_KEY_BUILDER.get(global, 
String.class);
+LOG.debug("profiler client: {}={}", PROFILER_ROW_KEY_BUILDER, 
rowKeyBuilderClass);
+
+// instantiate the RowKeyBuilder
+RowKeyBuilder builder = 
ReflectionUtils.createInstance(rowKeyBuilderClass);
+setSaltDivisor(global, builder);
+setPeriodDuration(global, builder);
--- End diff --

I don't really like how I go about setting the salt divisor and period 
duration on the `RowKeyBuilder`.  There are no methods in the `RowKeyBuilder` 
interface to do set these values.  I could add something like 
`RowKeyBuilder.setSaltDivisor`, but I was trying not to pollute that interface 
with variables like salt divisor that may not apply to all RowKeyBuilder 
implementations.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

2017-07-13 Thread nickwallen

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127329675
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/SaltyRowKeyBuilder.java
 ---
@@ -44,7 +46,17 @@
  * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
  * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
  * 
+ *
+ * This row key builder has no logic to decode a row key, nor is the row 
key generated by this builder
+ * easily decodable.  More specifically, the profile, entity, groups and 
period that make up the row key
+ * cannot be extracted from a previously generated row key.  This makes it 
difficult to answer questions
+ * like; What entities are included in this profile?  What is the period 
for this profile?  Use the
+ * DecodableRowKeyBuilder instead.
+ *
+ * @deprecated Replaced by DecodableRowKeyBuilder
+ * @see DecodableRowKeyBuilder
  */
+@Deprecated
 public class SaltyRowKeyBuilder implements RowKeyBuilder {
--- End diff --

I marked the old `RowKeyBuilder` as deprecated.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

2017-07-13 Thread nickwallen

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127326600
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/RowKeyBuilderFactory.java
 ---
@@ -0,0 +1,125 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.beanutils.PropertyUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.InvocationTargetException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_ROW_KEY_BUILDER;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * A Factory class that can create a RowKeyBuilder based on global 
property values.
+ */
+public class RowKeyBuilderFactory {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(RowKeyBuilderFactory.class);
+
+  /**
+   * Create a RowKeyBuilder.
+   * @param global The global properties.
+   * @return A RowKeyBuilder instantiated using the global property values.
+   */
+  public static RowKeyBuilder create(Map global) {
+String rowKeyBuilderClass = PROFILER_ROW_KEY_BUILDER.get(global, 
String.class);
+LOG.debug("profiler client: {}={}", PROFILER_ROW_KEY_BUILDER, 
rowKeyBuilderClass);
+
+// instantiate the RowKeyBuilder
+RowKeyBuilder builder = 
ReflectionUtils.createInstance(rowKeyBuilderClass);
+setSaltDivisor(global, builder);
+setPeriodDuration(global, builder);
--- End diff --

But I think this actually turned out worse, than the alternative of just 
adding `RowKeyBuilder.setSaltDivisor` and polluting the interface.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

2017-07-13 Thread nickwallen

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127325245
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/RowKeyBuilderFactory.java
 ---
@@ -0,0 +1,125 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.beanutils.PropertyUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.InvocationTargetException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_ROW_KEY_BUILDER;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * A Factory class that can create a RowKeyBuilder based on global 
property values.
+ */
+public class RowKeyBuilderFactory {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(RowKeyBuilderFactory.class);
+
+  /**
+   * Create a RowKeyBuilder.
+   * @param global The global properties.
+   * @return A RowKeyBuilder instantiated using the global property values.
+   */
+  public static RowKeyBuilder create(Map global) {
+String rowKeyBuilderClass = PROFILER_ROW_KEY_BUILDER.get(global, 
String.class);
+LOG.debug("profiler client: {}={}", PROFILER_ROW_KEY_BUILDER, 
rowKeyBuilderClass);
+
+// instantiate the RowKeyBuilder
+RowKeyBuilder builder = 
ReflectionUtils.createInstance(rowKeyBuilderClass);
+setSaltDivisor(global, builder);
+setPeriodDuration(global, builder);
--- End diff --

Here is the logic to instantiate a `RowKeyBuilder` that is used by the 
Profiler Client's `GetProfile`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

2017-07-13 Thread nickwallen

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127330773
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,382 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
--- End diff --

I added a `VERSION` field to the row key, hoping that this might help 
future changes to the `RowKeyBuilder`.  With this, I could potentially start to 
parse the row key and then choose the right `RowKeyBuilder` implementation; the 
one used to create the row key.  This would make row key changes seemless to 
users.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

2017-07-13 Thread nickwallen

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127329289
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/RowKeyBuilderFactory.java
 ---
@@ -0,0 +1,125 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.beanutils.PropertyUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.InvocationTargetException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_ROW_KEY_BUILDER;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * A Factory class that can create a RowKeyBuilder based on global 
property values.
+ */
+public class RowKeyBuilderFactory {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(RowKeyBuilderFactory.class);
+
+  /**
+   * Create a RowKeyBuilder.
+   * @param global The global properties.
+   * @return A RowKeyBuilder instantiated using the global property values.
+   */
+  public static RowKeyBuilder create(Map global) {
+String rowKeyBuilderClass = PROFILER_ROW_KEY_BUILDER.get(global, 
String.class);
+LOG.debug("profiler client: {}={}", PROFILER_ROW_KEY_BUILDER, 
rowKeyBuilderClass);
+
+// instantiate the RowKeyBuilder
+RowKeyBuilder builder = 
ReflectionUtils.createInstance(rowKeyBuilderClass);
+setSaltDivisor(global, builder);
+setPeriodDuration(global, builder);
+
+return builder;
+  }
+
+  /**
+   * Set the period duration on the RowKeyBuilder.
+   * @param global The global properties from Zk.
+   * @param builder The RowKeyBuilder implementation.
+   */
+  private static void setPeriodDuration(Map global, 
RowKeyBuilder builder) {
+
+// how long is the profile period?
+long duration = PROFILER_PERIOD.get(global, Long.class);
+LOG.debug("profiler client: {}={}", PROFILER_PERIOD, duration);
+
+// which units are used to define the profile period?
+String configuredUnits = PROFILER_PERIOD_UNITS.get(global, 
String.class);
+TimeUnit units = TimeUnit.valueOf(configuredUnits);
+LOG.debug("profiler client: {}={}", PROFILER_PERIOD_UNITS, units);
+
+// set the period duration
+final String periodDurationProperty = "periodDurationMillis";
+setProperty(builder, periodDurationProperty, units.toMillis(duration));
+  }
+
+  /**
+   * Set the salt divisor property on the RowKeyBuilder.
+   * @param global The global properties from Zk.
+   * @param builder The RowKeyBuilder implementation.
+   */
+  private static void setSaltDivisor(Map global, 
RowKeyBuilder builder) {
+
+// what is the salt divisor?
+Integer saltDivisor = PROFILER_SALT_DIVISOR.get(global, Integer.class);
+LOG.debug("profiler client: {}={}", PROFILER_SALT_DIVISOR, 
saltDivisor);
+
+final String saltDivisorProperty = "saltDivisor";
+setProperty(builder, saltDivisorProperty, saltDivisor);
--- End diff --

This basically sets the 'salt divisor' on any `RowKeyBuilder` that has a 
`saltDivisor` setter.  I really don't like this.  It is very hack-ish. I would 
love to use a simpler alternative.


---
If your project is set up for it, you can reply to this email and have your

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

2017-07-13 Thread nickwallen

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127328660
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java
 ---
@@ -216,21 +211,7 @@ private ColumnBuilder getColumnBuilder(Map global) {
* @param global The global configuration.
*/
   private RowKeyBuilder getRowKeyBuilder(Map global) {
-
-// how long is the profile period?
-long duration = PROFILER_PERIOD.get(global, Long.class);
-LOG.debug("profiler client: {}={}", PROFILER_PERIOD, duration);
-
-// which units are used to define the profile period?
-String configuredUnits = PROFILER_PERIOD_UNITS.get(global, 
String.class);
-TimeUnit units = TimeUnit.valueOf(configuredUnits);
-LOG.debug("profiler client: {}={}", PROFILER_PERIOD_UNITS, units);
-
-// what is the salt divisor?
-Integer saltDivisor = PROFILER_SALT_DIVISOR.get(global, Integer.class);
-LOG.debug("profiler client: {}={}", PROFILER_SALT_DIVISOR, 
saltDivisor);
-
-return new SaltyRowKeyBuilder(saltDivisor, duration, units);
+return RowKeyBuilderFactory.create(global);
--- End diff --

This is where we need to instantiate the `RowKeyBuilder` for the Profiler 
Client API.  Like I will discuss in another thread, the logic got complex and 
kind of nasty so I encapsulated it in its own `RowKeyBuilderFactory`.  See that 
class for a further discussion as to why it is kind of nasty.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

2017-07-13 Thread nickwallen

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127329548
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,382 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
--- End diff --

The new `RowKeyBuilder` implementation that is decodable.  Everyone should 
just use this, but the older implementation is left for backwards compatibility.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

2017-07-13 Thread nickwallen

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127327126
  
--- Diff: 
metron-analytics/metron-profiler/src/main/flux/profiler/remote.yaml ---
@@ -29,7 +29,7 @@ components:
 - name: "saltDivisor"
--- End diff --

Notice that the legacy `RowKeyBuilder`, the `SaltyRowKeyBuilder`, is still 
the default.  If a user wants to use the new `RowKeyBuilder` then they need to 
change the flux file here and specify 
`org.apache.metron.profiler.hbase.DecodableRowKeyBuilder`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #642: METRON-984 Stellar functions to decode encoded fields or ...

2017-07-13 Thread ottobackwards

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/642
  
@cestella, I don't see my functions listed in the config ui, but they are 
in the shell.  That doesn't seem right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #652: METRON-1039: Add ZIP function to Stellar

2017-07-13 Thread jjmeyer0

Github user jjmeyer0 commented on the issue:

https://github.com/apache/metron/pull/652
  
@cestella I was wondering what you think about having a zip function that 
we could pass a lambda to. It could potentially be nicer to read than combining 
a `ZIP` with a `REDUCE`. 

```
ZIP_WITH([1, 2, 3], [1, 2, 3], (a, b) -> a + b) == [2, 4, 6]
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #653: METRON-1040 Create Installation Instructions for t...

2017-07-13 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/653


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #653: METRON-1040 Create Installation Instructions for the Prof...

2017-07-13 Thread nickwallen

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/653
  
I have been able to follow these steps successfully in Full Dev 
environment.  Looking good to me.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #481: METRON-322 Global Batching and Flushing

2017-07-13 Thread mattf-horton

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/481
  
Before committing, I'm going to retest to make sure I'm not missing the 
problems @ottobackwards saw.  @dlyle65535 , also interested in your feedback if 
you have time.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #623: METRON-1003 ParserUtil parses dates incorrect

2017-07-13 Thread justinleet

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/623
  
@bjigmp I'm fine with doing that later.  If you add the comment I asked for 
(honestly mostly because I personally had no idea what that method expected 
until I looked at the method more) I'm +1, and we can hopefully also get 
https://github.com/apache/metron/pull/528 in soon, too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Relocate Docker

2017-07-13 Thread zeo...@gmail.com

I agree to moving it to a contrib or contrib-like area.

Jon

On Thu, Jul 13, 2017 at 12:38 PM Kyle Richardson 
wrote:

> I completely support the idea of moving metron-docker down in the tree. I
> do like the idea of a contrib/ area for things like this that are not as
> frequently updated or maintained. Are there any other pieces of the code
> base that would fit into this type of area?
>
> -Kyle
>
> On Thu, Jul 13, 2017 at 12:30 PM Laurens Vets  wrote:
>
> > On 2017-07-13 09:04, Nick Allen wrote:
> > > Having metron-docker at the top-level of the project seems to catch the
> > > attention of new users.  Some then start using metron-docker to
> > > explore/try-out/demo Metron.
> > >
> > > The metron-docker code that we have is not well-suited for this
> > > purpose.
> > > It is only really useful for development.  It is not regularly tested
> > > and
> > > maintained like our Vagrant environment.
> > >
> > > I am proposing that we move the top-level "metron-docker" directory to
> > > live
> > > under "metron-deployment" to avoid confusing new users.  This also
> > > seems
> > > more logical to me. I would also pair this move with some better
> > > "Getting
> > > Started" steps for new users in the top-level README.
> > >
> > > cd $METRON_HOME
> > > mv metron-docker metron-deployment/docker
> > >
> > >
> > > Do you agree/disagree?  Is there a better solution?
> >
> > Maybe under "Other Examples"?
> >
>
-- 

Jon

Re: [DISCUSS] Relocate Docker

2017-07-13 Thread Nick Allen

I could be convinced to get onboard with a contrib (or metron-contrib to
follow convention?) top-level directory.  My only concern is that it might
become a dumping ground of random stuff.  Maybe that concern is not
warranted.

On Thu, Jul 13, 2017 at 12:38 PM, Kyle Richardson  wrote:

> I completely support the idea of moving metron-docker down in the tree. I
> do like the idea of a contrib/ area for things like this that are not as
> frequently updated or maintained. Are there any other pieces of the code
> base that would fit into this type of area?
>
> -Kyle
>
> On Thu, Jul 13, 2017 at 12:30 PM Laurens Vets  wrote:
>
> > On 2017-07-13 09:04, Nick Allen wrote:
> > > Having metron-docker at the top-level of the project seems to catch the
> > > attention of new users.  Some then start using metron-docker to
> > > explore/try-out/demo Metron.
> > >
> > > The metron-docker code that we have is not well-suited for this
> > > purpose.
> > > It is only really useful for development.  It is not regularly tested
> > > and
> > > maintained like our Vagrant environment.
> > >
> > > I am proposing that we move the top-level "metron-docker" directory to
> > > live
> > > under "metron-deployment" to avoid confusing new users.  This also
> > > seems
> > > more logical to me. I would also pair this move with some better
> > > "Getting
> > > Started" steps for new users in the top-level README.
> > >
> > > cd $METRON_HOME
> > > mv metron-docker metron-deployment/docker
> > >
> > >
> > > Do you agree/disagree?  Is there a better solution?
> >
> > Maybe under "Other Examples"?
> >
>

[GitHub] metron issue #623: METRON-1003 ParserUtil parses dates incorrect

2017-07-13 Thread bjigmp

Github user bjigmp commented on the issue:

https://github.com/apache/metron/pull/623
  
@justinleet I absolutely agree with you that this method must be rewritten 
and probably moved to FireEye.
But my plan was to fix it/write unit test and then refactor it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #651: METRON-1037 Added POWER function

2017-07-13 Thread mattf-horton

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/651
  
Har, I give up.  POWER it is.  I agree would be good to use @cestella 's 
new framework, and see above review comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #651: METRON-1037 Added POWER function

2017-07-13 Thread mattf-horton

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/651#discussion_r127277799
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/MathFunctions.java
 ---
@@ -60,4 +60,38 @@ public boolean isInitialized() {
 }
   }
 
+  @Stellar(name="POWER"
+   ,description="Raises the number to a given power"
+   ,params = {
+  "number - the number",
+  "power - the exponent to raise the number to"
+ }
+   , returns="The power-th power of the number passed in."
--- End diff --

ditto: "The value of number to the given power."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #651: METRON-1037 Added POWER function

2017-07-13 Thread mattf-horton

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/651#discussion_r127279766
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/MathFunctionsTest.java
 ---
@@ -44,4 +44,16 @@ public void testAbs() {
 Assert.assertEquals((Double)run("ABS(value)", ImmutableMap.of("value", 
-10.5)), 10.5, 1e-7);
   }
 
+  @Test
+  public void testPower() {
+Assert.assertEquals((Double)run("POWER(e,b)", ImmutableMap.of("e", 
2.0, "b", 2.0)), 4.0, 1e-7 );
--- End diff --

Please put expected value first, test result value second, in 
assertEquals() calls.  Otherwise reported failures look funny.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #651: METRON-1037 Added POWER function

2017-07-13 Thread mattf-horton

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/651#discussion_r127279153
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/MathFunctionsTest.java
 ---
@@ -44,4 +44,16 @@ public void testAbs() {
 Assert.assertEquals((Double)run("ABS(value)", ImmutableMap.of("value", 
-10.5)), 10.5, 1e-7);
   }
 
+  @Test
+  public void testPower() {
+Assert.assertEquals((Double)run("POWER(e,b)", ImmutableMap.of("e", 
2.0, "b", 2.0)), 4.0, 1e-7 );
+Assert.assertTrue(Double.isNaN((Double)run("ABS(value)", 
ImmutableMap.of("e", Double.NaN, "b", 2.0;
+Assert.assertTrue(Double.isNaN((Double)run("ABS(value)", 
ImmutableMap.of("e", 2.0, "b", Double.NaN;
+Assert.assertTrue(Double.isNaN((Double)run("ABS(value)", 
ImmutableMap.of("e", Double.NaN, "b", Double.NaN;
+
--- End diff --

Were these 3 lines supposed to be POWER(e,b) invocations rather than 
ABS(value)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #651: METRON-1037 Added POWER function

2017-07-13 Thread mattf-horton

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/651#discussion_r127277470
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -518,6 +519,13 @@ In the core language functions, we support basic 
functional programming primitiv
 * dateTime - The datetime as a long representing the milliseconds 
since unix epoch
   * Returns: The current month (0-based).
 
+### `POWER`
+  * Description: Raises the number to a given power.
+  * Input:
+* number - the number
+* power - the exponent to raise the number to
+  * Returns: The power-th power of the number passed in.
--- End diff --

Suggest: Returns: The value of number to the given power.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #642: METRON-984 Stellar functions to decode encoded fields or ...

2017-07-13 Thread ottobackwards

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/642
  
Will do


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #651: METRON-1037 Added POWER function

2017-07-13 Thread cestella

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/651
  
@mattf-horton Actually, `**` and `math.exp()` are different;  the first 
being an arbitrary power function and the later presuming that we're taking a 
power of `e` (i.e. `e**x == exp(x)` ).  I did add `EXP` (as in `e**x`) in #650


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #653: METRON-1040 Create Installation Instructions for the Prof...

2017-07-13 Thread cestella

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/653
  
Exactly


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #653: METRON-1040 Create Installation Instructions for the Prof...

2017-07-13 Thread nickwallen

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/653
  
Ah, sure.  Creating the HBase table is described in the "Getting Started" 
section, but that does seem to make more sense under "Installation" now.  Will 
do. Thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #653: METRON-1040 Create Installation Instructions for the Prof...

2017-07-13 Thread nickwallen

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/653
  
Oh, like adjust the following in `profiler.properties`?

```
...
kafka.zk=node1:2181
kafka.broker=node1:6667
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #651: METRON-1037 Added POWER function

2017-07-13 Thread mattf-horton

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/651
  
@simonellistonball , you're right, '**' would have been far more 
appropriate given what most languages do.  And I find your argument convincing 
that the function form will be more familiar to many.
@cestella , python allows both '**' and "math.exp()".  Java also uses 
"Math.exp()".  So I recommend we go with EXP().  Did I see in your comment in 
#650 that you've now added EXP to your PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #653: METRON-1040 Create Installation Instructions for the Prof...

2017-07-13 Thread nickwallen

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/653
  
@cestella I am not following.  Can you describe that a little more?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #653: METRON-1040 Create Installation Instructions for the Prof...

2017-07-13 Thread cestella

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/653
  
Cool, can we get a blurb about adjusting configs to conform to your system, 
etc. at the end of this?  If so, I'm +1.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #636: METRON-1022: Elasticsearch REST endpoint

2017-07-13 Thread cestella

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/636
  
Bear with me, @merrimanr, I am going to submit a PR with the DAO 
abstraction I was talking about so we can hash it out.  I started it in 
collaboration with @justinleet to ensure the ideas in the mutating indexed data 
and the meta alerts discuss threads made sense.  I'll just contribute the core 
abstraction here.  Unfortunately I won't be able to get around to it until 
Tuesday (vacation and all ;).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #651: METRON-1037 Added POWER function

2017-07-13 Thread simonellistonball

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/651
  
Yes, I think I'll just refactor this around #650 when that's committed to 
keep the workflow simple.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #651: METRON-1037 Added POWER function

2017-07-13 Thread simonellistonball

Github user simonellistonball closed the pull request at:

https://github.com/apache/metron/pull/651


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #649: METRON-1035 Added SUM to the rules triage aggregation doc...

2017-07-13 Thread cestella

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/649
  
+1 by inspection, nice work


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Relocate Docker

2017-07-13 Thread Kyle Richardson

I completely support the idea of moving metron-docker down in the tree. I
do like the idea of a contrib/ area for things like this that are not as
frequently updated or maintained. Are there any other pieces of the code
base that would fit into this type of area?

-Kyle

On Thu, Jul 13, 2017 at 12:30 PM Laurens Vets  wrote:

> On 2017-07-13 09:04, Nick Allen wrote:
> > Having metron-docker at the top-level of the project seems to catch the
> > attention of new users.  Some then start using metron-docker to
> > explore/try-out/demo Metron.
> >
> > The metron-docker code that we have is not well-suited for this
> > purpose.
> > It is only really useful for development.  It is not regularly tested
> > and
> > maintained like our Vagrant environment.
> >
> > I am proposing that we move the top-level "metron-docker" directory to
> > live
> > under "metron-deployment" to avoid confusing new users.  This also
> > seems
> > more logical to me. I would also pair this move with some better
> > "Getting
> > Started" steps for new users in the top-level README.
> >
> > cd $METRON_HOME
> > mv metron-docker metron-deployment/docker
> >
> >
> > Do you agree/disagree?  Is there a better solution?
>
> Maybe under "Other Examples"?
>

Re: [DISCUSS] Relocate Docker

2017-07-13 Thread Laurens Vets


On 2017-07-13 09:04, Nick Allen wrote:

Having metron-docker at the top-level of the project seems to catch the
attention of new users.  Some then start using metron-docker to
explore/try-out/demo Metron.

The metron-docker code that we have is not well-suited for this 
purpose.
It is only really useful for development.  It is not regularly tested 
and

maintained like our Vagrant environment.

I am proposing that we move the top-level "metron-docker" directory to 
live
under "metron-deployment" to avoid confusing new users.  This also 
seems
more logical to me. I would also pair this move with some better 
"Getting

Started" steps for new users in the top-level README.

cd $METRON_HOME
mv metron-docker metron-deployment/docker


Do you agree/disagree?  Is there a better solution?


Maybe under "Other Examples"?

[GitHub] metron issue #651: METRON-1037 Added POWER function

2017-07-13 Thread simonellistonball

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/651
  
I get the point @mattf-horton and thought about it, but went this way for 
familiarity's sake, a lot of the likely authors of stellar statements are 
security analysts who will be more familiar with languages like SQL or Excel 
functions, or even python than languages which use the ^ operator (off the top 
of my head:  urgh... just tried python, scala, java, c, c++, javascript, can't 
actually think of one that does, though some have **)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #651: METRON-1037 Added POWER function

2017-07-13 Thread cestella

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/651#discussion_r127263511
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/MathFunctions.java
 ---
@@ -60,4 +60,38 @@ public boolean isInitialized() {
 }
   }
 
+  @Stellar(name="POWER"
--- End diff --

can we call it POW?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #650: METRON-1038: Stellar should have a better collection of b...

2017-07-13 Thread cestella

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/650
  
Ok, I broke this abstraction out a bit and made it easier to add 
non-single-arg functions (a la POW, which is a separate PR by 
@simonellistonball ).  I also went ahead and added `ROUND` and `EXP`.  I looked 
through the remainder of the math functions and honestly, I didn't see a strong 
reason to include any of the rest of them immediately.  I'd RATHER them come in 
via separate PRs with some justification of use.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #651: METRON-1037 Added POWER function

2017-07-13 Thread mattf-horton

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/651
  
I would rather see this implemented as a `^` math operator rather than a 
function, even though Java doesn't.  Opinions?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Relocate Docker

2017-07-13 Thread Otto Fowler

I think that if it is not a ‘build breaking dependency’ it should be in a
/contrib area and not in deployment.



On July 13, 2017 at 12:05:00, Nick Allen (n...@nickallen.org) wrote:

Having metron-docker at the top-level of the project seems to catch the
attention of new users. Some then start using metron-docker to
explore/try-out/demo Metron.

The metron-docker code that we have is not well-suited for this purpose.
It is only really useful for development. It is not regularly tested and
maintained like our Vagrant environment.

I am proposing that we move the top-level "metron-docker" directory to live
under "metron-deployment" to avoid confusing new users. This also seems
more logical to me. I would also pair this move with some better "Getting
Started" steps for new users in the top-level README.

cd $METRON_HOME
mv metron-docker metron-deployment/docker


Do you agree/disagree? Is there a better solution?

[DISCUSS] Relocate Docker

2017-07-13 Thread Nick Allen

Having metron-docker at the top-level of the project seems to catch the
attention of new users.  Some then start using metron-docker to
explore/try-out/demo Metron.

The metron-docker code that we have is not well-suited for this purpose.
It is only really useful for development.  It is not regularly tested and
maintained like our Vagrant environment.

I am proposing that we move the top-level "metron-docker" directory to live
under "metron-deployment" to avoid confusing new users.  This also seems
more logical to me. I would also pair this move with some better "Getting
Started" steps for new users in the top-level README.

cd $METRON_HOME
mv metron-docker metron-deployment/docker


Do you agree/disagree?  Is there a better solution?

[GitHub] metron pull request #652: METRON-1039: Add ZIP function to Stellar

2017-07-13 Thread cestella

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/652#discussion_r127247191
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/FunctionalFunctionsTest.java
 ---
@@ -24,13 +24,124 @@
 import org.junit.Assert;
 import org.junit.Test;
 
+import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 
 import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
 
 public class FunctionalFunctionsTest {
 
   @Test
+  public void testZipLongest_boundary() {
+for (String expr : ImmutableList.of( "ZIP_LONGEST()"
+, "ZIP_LONGEST( null, null )"
+, "ZIP_LONGEST( [], null )"
+, "ZIP_LONGEST( [], [] )"
+, "ZIP_LONGEST( null, [] )"
+)
+)
+{
+  List o = (List) run(expr, new 
HashMap<>());
+  Assert.assertEquals(0, o.size());
+}
+  }
+
+  @Test
+  public void testZip_longest() {
+Map variables = ImmutableMap.of(
+"list1" , ImmutableList.of(1, 2, 3)
+,"list2", ImmutableList.of(4, 5, 6, 7)
+);
+for (String expr : ImmutableList.of( "ZIP_LONGEST(list1, list2)"
+, "ZIP_LONGEST( [1, 2, 3], [4, 5, 6, 7] )"
+)
+)
+{
+  List o = (List) run(expr, variables);
+  Assert.assertEquals(4, o.size());
+  for (int i = 0; i < 3; ++i) {
+List l = o.get(i);
+Assert.assertEquals(2, l.size());
+Assert.assertEquals(i+1, l.get(0));
+Assert.assertEquals(i+4, l.get(1));
+  }
+  {
+int i = 3;
+List l = o.get(i);
+Assert.assertEquals(2, l.size());
+Assert.assertNull(l.get(0));
+Assert.assertEquals(i+4, l.get(1));
+  }
+}
+
+
+for (String expr : ImmutableList.of(
+ "REDUCE(ZIP_LONGEST(list2, list1), (s, x) -> s + GET_FIRST(x) 
* GET_LAST(x), 0)"
+, "REDUCE(ZIP_LONGEST( [1, 2, 3], [4, 5, 6, 7] ), (s, x) -> s 
+ GET_FIRST(x) * GET_LAST(x), 0)"
+, "REDUCE(ZIP_LONGEST(list1, list2), (s, x) -> s + 
GET_FIRST(x) * GET_LAST(x), 0)" //this works because stellar treats nulls as 0 
in arithmetic operations.
+, "REDUCE(ZIP_LONGEST(list1, list2), (s, x) -> s + 
(GET_FIRST(x) == null?0:GET_FIRST(x)) * (GET_LAST(x) == null?0:GET_LAST(x)), 
0)" //with proper guarding NOT assuming stellar peculiarities
+)
+)
+{
+  int o = (int) run(expr, variables);
+  Assert.assertEquals(1*4 + 2*5 + 3*6, o, 1e-7);
--- End diff --

Entirely reasonable conclusion, though @simonellistonball Good to think it 
through.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #652: METRON-1039: Add ZIP function to Stellar

2017-07-13 Thread ottobackwards

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/652#discussion_r127245708
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/FunctionalFunctionsTest.java
 ---
@@ -24,13 +24,124 @@
 import org.junit.Assert;
 import org.junit.Test;
 
+import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 
 import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
 
 public class FunctionalFunctionsTest {
 
   @Test
+  public void testZipLongest_boundary() {
+for (String expr : ImmutableList.of( "ZIP_LONGEST()"
+, "ZIP_LONGEST( null, null )"
+, "ZIP_LONGEST( [], null )"
+, "ZIP_LONGEST( [], [] )"
+, "ZIP_LONGEST( null, [] )"
+)
+)
+{
+  List o = (List) run(expr, new 
HashMap<>());
+  Assert.assertEquals(0, o.size());
+}
+  }
+
+  @Test
+  public void testZip_longest() {
+Map variables = ImmutableMap.of(
+"list1" , ImmutableList.of(1, 2, 3)
+,"list2", ImmutableList.of(4, 5, 6, 7)
+);
+for (String expr : ImmutableList.of( "ZIP_LONGEST(list1, list2)"
+, "ZIP_LONGEST( [1, 2, 3], [4, 5, 6, 7] )"
+)
+)
+{
+  List o = (List) run(expr, variables);
+  Assert.assertEquals(4, o.size());
+  for (int i = 0; i < 3; ++i) {
+List l = o.get(i);
+Assert.assertEquals(2, l.size());
+Assert.assertEquals(i+1, l.get(0));
+Assert.assertEquals(i+4, l.get(1));
+  }
+  {
+int i = 3;
+List l = o.get(i);
+Assert.assertEquals(2, l.size());
+Assert.assertNull(l.get(0));
+Assert.assertEquals(i+4, l.get(1));
+  }
+}
+
+
+for (String expr : ImmutableList.of(
+ "REDUCE(ZIP_LONGEST(list2, list1), (s, x) -> s + GET_FIRST(x) 
* GET_LAST(x), 0)"
+, "REDUCE(ZIP_LONGEST( [1, 2, 3], [4, 5, 6, 7] ), (s, x) -> s 
+ GET_FIRST(x) * GET_LAST(x), 0)"
+, "REDUCE(ZIP_LONGEST(list1, list2), (s, x) -> s + 
GET_FIRST(x) * GET_LAST(x), 0)" //this works because stellar treats nulls as 0 
in arithmetic operations.
+, "REDUCE(ZIP_LONGEST(list1, list2), (s, x) -> s + 
(GET_FIRST(x) == null?0:GET_FIRST(x)) * (GET_LAST(x) == null?0:GET_LAST(x)), 
0)" //with proper guarding NOT assuming stellar peculiarities
+)
+)
+{
+  int o = (int) run(expr, variables);
+  Assert.assertEquals(1*4 + 2*5 + 3*6, o, 1e-7);
--- End diff --

and don't call him shirly


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #652: METRON-1039: Add ZIP function to Stellar

2017-07-13 Thread ottobackwards

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/652#discussion_r127245624
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/FunctionalFunctionsTest.java
 ---
@@ -24,13 +24,124 @@
 import org.junit.Assert;
 import org.junit.Test;
 
+import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 
 import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
 
 public class FunctionalFunctionsTest {
 
   @Test
+  public void testZipLongest_boundary() {
+for (String expr : ImmutableList.of( "ZIP_LONGEST()"
+, "ZIP_LONGEST( null, null )"
+, "ZIP_LONGEST( [], null )"
+, "ZIP_LONGEST( [], [] )"
+, "ZIP_LONGEST( null, [] )"
+)
+)
+{
+  List o = (List) run(expr, new 
HashMap<>());
+  Assert.assertEquals(0, o.size());
+}
+  }
+
+  @Test
+  public void testZip_longest() {
+Map variables = ImmutableMap.of(
+"list1" , ImmutableList.of(1, 2, 3)
+,"list2", ImmutableList.of(4, 5, 6, 7)
+);
+for (String expr : ImmutableList.of( "ZIP_LONGEST(list1, list2)"
+, "ZIP_LONGEST( [1, 2, 3], [4, 5, 6, 7] )"
+)
+)
+{
+  List o = (List) run(expr, variables);
+  Assert.assertEquals(4, o.size());
+  for (int i = 0; i < 3; ++i) {
+List l = o.get(i);
+Assert.assertEquals(2, l.size());
+Assert.assertEquals(i+1, l.get(0));
+Assert.assertEquals(i+4, l.get(1));
+  }
+  {
+int i = 3;
+List l = o.get(i);
+Assert.assertEquals(2, l.size());
+Assert.assertNull(l.get(0));
+Assert.assertEquals(i+4, l.get(1));
+  }
+}
+
+
+for (String expr : ImmutableList.of(
+ "REDUCE(ZIP_LONGEST(list2, list1), (s, x) -> s + GET_FIRST(x) 
* GET_LAST(x), 0)"
+, "REDUCE(ZIP_LONGEST( [1, 2, 3], [4, 5, 6, 7] ), (s, x) -> s 
+ GET_FIRST(x) * GET_LAST(x), 0)"
+, "REDUCE(ZIP_LONGEST(list1, list2), (s, x) -> s + 
GET_FIRST(x) * GET_LAST(x), 0)" //this works because stellar treats nulls as 0 
in arithmetic operations.
+, "REDUCE(ZIP_LONGEST(list1, list2), (s, x) -> s + 
(GET_FIRST(x) == null?0:GET_FIRST(x)) * (GET_LAST(x) == null?0:GET_LAST(x)), 
0)" //with proper guarding NOT assuming stellar peculiarities
+)
+)
+{
+  int o = (int) run(expr, variables);
+  Assert.assertEquals(1*4 + 2*5 + 3*6, o, 1e-7);
--- End diff --

math fight!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

2017-07-13 Thread ottobackwards

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/650#discussion_r127245372
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/MathFunctions.java
 ---
@@ -25,17 +25,39 @@
 import org.apache.metron.stellar.dsl.StellarFunction;
 
 import java.util.List;
+import java.util.function.Function;
 
 public class MathFunctions {
 
-  @Stellar(name="ABS"
-  ,description="Returns the absolute value of a number."
-  ,params = {
-"number - The number to take the absolute value of"
-}
-  , returns="The absolute value of the number passed in."
-  )
-  public static class Abs implements StellarFunction {
+  private enum SingleArgMathFunctions implements Function {
+ABS(d -> Math.abs(d)),
+CEIL(d -> Math.ceil(d)),
+COS(d -> Math.cos(d)),
+FLOOR(d -> Math.floor(d)),
+LOG10(d -> Math.log10(d)),
+LOG2(d -> Math.log(d)/Math.log(2)),
+LN(d -> Math.log(d)),
+SIN(d -> Math.sin(d)),
+SQRT(d -> Math.sqrt(d)),
+TAN(d -> Math.tan(d)),
+;
+
--- End diff --

I agree.  I think that in general when we are adapting or exposing a set of 
functionality  from java or from apache-commons  we should state our intent 
explicitly for the record.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #652: METRON-1039: Add ZIP function to Stellar

2017-07-13 Thread simonellistonball

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/652#discussion_r127244811
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/FunctionalFunctionsTest.java
 ---
@@ -24,13 +24,124 @@
 import org.junit.Assert;
 import org.junit.Test;
 
+import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 
 import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
 
 public class FunctionalFunctionsTest {
 
   @Test
+  public void testZipLongest_boundary() {
+for (String expr : ImmutableList.of( "ZIP_LONGEST()"
+, "ZIP_LONGEST( null, null )"
+, "ZIP_LONGEST( [], null )"
+, "ZIP_LONGEST( [], [] )"
+, "ZIP_LONGEST( null, [] )"
+)
+)
+{
+  List o = (List) run(expr, new 
HashMap<>());
+  Assert.assertEquals(0, o.size());
+}
+  }
+
+  @Test
+  public void testZip_longest() {
+Map variables = ImmutableMap.of(
+"list1" , ImmutableList.of(1, 2, 3)
+,"list2", ImmutableList.of(4, 5, 6, 7)
+);
+for (String expr : ImmutableList.of( "ZIP_LONGEST(list1, list2)"
+, "ZIP_LONGEST( [1, 2, 3], [4, 5, 6, 7] )"
+)
+)
+{
+  List o = (List) run(expr, variables);
+  Assert.assertEquals(4, o.size());
+  for (int i = 0; i < 3; ++i) {
+List l = o.get(i);
+Assert.assertEquals(2, l.size());
+Assert.assertEquals(i+1, l.get(0));
+Assert.assertEquals(i+4, l.get(1));
+  }
+  {
+int i = 3;
+List l = o.get(i);
+Assert.assertEquals(2, l.size());
+Assert.assertNull(l.get(0));
+Assert.assertEquals(i+4, l.get(1));
+  }
+}
+
+
+for (String expr : ImmutableList.of(
+ "REDUCE(ZIP_LONGEST(list2, list1), (s, x) -> s + GET_FIRST(x) 
* GET_LAST(x), 0)"
+, "REDUCE(ZIP_LONGEST( [1, 2, 3], [4, 5, 6, 7] ), (s, x) -> s 
+ GET_FIRST(x) * GET_LAST(x), 0)"
+, "REDUCE(ZIP_LONGEST(list1, list2), (s, x) -> s + 
GET_FIRST(x) * GET_LAST(x), 0)" //this works because stellar treats nulls as 0 
in arithmetic operations.
+, "REDUCE(ZIP_LONGEST(list1, list2), (s, x) -> s + 
(GET_FIRST(x) == null?0:GET_FIRST(x)) * (GET_LAST(x) == null?0:GET_LAST(x)), 
0)" //with proper guarding NOT assuming stellar peculiarities
+)
+)
+{
+  int o = (int) run(expr, variables);
+  Assert.assertEquals(1*4 + 2*5 + 3*6, o, 1e-7);
--- End diff --

good point, i should probably have looked at the numbers.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

2017-07-13 Thread cestella

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/650#discussion_r127242712
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/MathFunctions.java
 ---
@@ -25,17 +25,39 @@
 import org.apache.metron.stellar.dsl.StellarFunction;
 
 import java.util.List;
+import java.util.function.Function;
 
 public class MathFunctions {
 
-  @Stellar(name="ABS"
-  ,description="Returns the absolute value of a number."
-  ,params = {
-"number - The number to take the absolute value of"
-}
-  , returns="The absolute value of the number passed in."
-  )
-  public static class Abs implements StellarFunction {
+  private enum SingleArgMathFunctions implements Function {
+ABS(d -> Math.abs(d)),
+CEIL(d -> Math.ceil(d)),
+COS(d -> Math.cos(d)),
+FLOOR(d -> Math.floor(d)),
+LOG10(d -> Math.log10(d)),
+LOG2(d -> Math.log(d)/Math.log(2)),
+LN(d -> Math.log(d)),
+SIN(d -> Math.sin(d)),
+SQRT(d -> Math.sqrt(d)),
+TAN(d -> Math.tan(d)),
+;
+
--- End diff --

I think they're all applicable, most of the remainder aren't as useful to 
Metron, IMO, but I would be willing to go through and add them if we feel that 
there is a need for completeness.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

2017-07-13 Thread cestella

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/650#discussion_r127242100
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/MathFunctions.java
 ---
@@ -25,17 +25,39 @@
 import org.apache.metron.stellar.dsl.StellarFunction;
 
 import java.util.List;
+import java.util.function.Function;
 
 public class MathFunctions {
 
-  @Stellar(name="ABS"
-  ,description="Returns the absolute value of a number."
-  ,params = {
-"number - The number to take the absolute value of"
-}
-  , returns="The absolute value of the number passed in."
-  )
-  public static class Abs implements StellarFunction {
+  private enum SingleArgMathFunctions implements Function {
+ABS(d -> Math.abs(d)),
+CEIL(d -> Math.ceil(d)),
+COS(d -> Math.cos(d)),
+FLOOR(d -> Math.floor(d)),
+LOG10(d -> Math.log10(d)),
+LOG2(d -> Math.log(d)/Math.log(2)),
+LN(d -> Math.log(d)),
+SIN(d -> Math.sin(d)),
+SQRT(d -> Math.sqrt(d)),
+TAN(d -> Math.tan(d)),
+;
+
--- End diff --

Honestly, I just added the ones that seem to be the most pressingly missing 
and that we could not recreate using primitives.  Did you have any functions 
that you wanted in there?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #652: METRON-1039: Add ZIP function to Stellar

2017-07-13 Thread simonellistonball

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/652#discussion_r127239254
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/FunctionalFunctionsTest.java
 ---
@@ -24,13 +24,124 @@
 import org.junit.Assert;
 import org.junit.Test;
 
+import java.util.ArrayList;
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 
 import static 
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
 
 public class FunctionalFunctionsTest {
 
   @Test
+  public void testZipLongest_boundary() {
+for (String expr : ImmutableList.of( "ZIP_LONGEST()"
+, "ZIP_LONGEST( null, null )"
+, "ZIP_LONGEST( [], null )"
+, "ZIP_LONGEST( [], [] )"
+, "ZIP_LONGEST( null, [] )"
+)
+)
+{
+  List o = (List) run(expr, new 
HashMap<>());
+  Assert.assertEquals(0, o.size());
+}
+  }
+
+  @Test
+  public void testZip_longest() {
+Map variables = ImmutableMap.of(
+"list1" , ImmutableList.of(1, 2, 3)
+,"list2", ImmutableList.of(4, 5, 6, 7)
+);
+for (String expr : ImmutableList.of( "ZIP_LONGEST(list1, list2)"
+, "ZIP_LONGEST( [1, 2, 3], [4, 5, 6, 7] )"
+)
+)
+{
+  List o = (List) run(expr, variables);
+  Assert.assertEquals(4, o.size());
+  for (int i = 0; i < 3; ++i) {
+List l = o.get(i);
+Assert.assertEquals(2, l.size());
+Assert.assertEquals(i+1, l.get(0));
+Assert.assertEquals(i+4, l.get(1));
+  }
+  {
+int i = 3;
+List l = o.get(i);
+Assert.assertEquals(2, l.size());
+Assert.assertNull(l.get(0));
+Assert.assertEquals(i+4, l.get(1));
+  }
+}
+
+
+for (String expr : ImmutableList.of(
+ "REDUCE(ZIP_LONGEST(list2, list1), (s, x) -> s + GET_FIRST(x) 
* GET_LAST(x), 0)"
+, "REDUCE(ZIP_LONGEST( [1, 2, 3], [4, 5, 6, 7] ), (s, x) -> s 
+ GET_FIRST(x) * GET_LAST(x), 0)"
+, "REDUCE(ZIP_LONGEST(list1, list2), (s, x) -> s + 
GET_FIRST(x) * GET_LAST(x), 0)" //this works because stellar treats nulls as 0 
in arithmetic operations.
+, "REDUCE(ZIP_LONGEST(list1, list2), (s, x) -> s + 
(GET_FIRST(x) == null?0:GET_FIRST(x)) * (GET_LAST(x) == null?0:GET_LAST(x)), 
0)" //with proper guarding NOT assuming stellar peculiarities
+)
+)
+{
+  int o = (int) run(expr, variables);
+  Assert.assertEquals(1*4 + 2*5 + 3*6, o, 1e-7);
--- End diff --

surely this does not actually test the structure is correct. I could get 
the same asserted result from zip(a, reversed(b)). I think we should have tests 
around defined ordering of pairs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #652: METRON-1039: Add ZIP function to Stellar

2017-07-13 Thread nickwallen

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/652
  
Do we have good unit tests for the function's behavior with empty lists, 
nulls and other boundary conditions?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #653: METRON-1040 Create Installation Instructions for t...

2017-07-13 Thread nickwallen

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/653

METRON-1040 Create Installation Instructions for the Profiler

Created additional instructions for installing the Profiler.  

## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1040

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/653.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #653


commit 1e6e04ac9ff09831abb5e77707336bb52e9704fd
Author: Nick Allen 
Date:   2017-07-13T14:32:20Z

METRON-1040 Create Installation Instructions for the Profiler




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #652: METRON-1039: Add ZIP function to Stellar

2017-07-13 Thread ottobackwards

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/652
  
+1 - built + test travis style, ran stellar cli.
Nice work @cestella 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #652: METRON-1039: Add ZIP function to Stellar

2017-07-13 Thread simonellistonball

Github user simonellistonball commented on a diff in the pull request:

https://github.com/apache/metron/pull/652#discussion_r127231248
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -711,6 +713,18 @@ In the core language functions, we support basic 
functional programming primitiv
 * dateTime - The datetime as a long representing the milliseconds 
since unix epoch
   * Returns: The current year
 
+### `ZIP`
+  * Description: Zips lists into a single list where the ith element is an 
list containing the ith items from the constituent lists.
--- End diff --

might be kinder to link to the python docs on zip, or at least an example 
that isn't in lisp :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #652: METRON-1039: Add ZIP function to Stellar

2017-07-13 Thread ottobackwards

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/652#discussion_r127230597
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -711,6 +713,18 @@ In the core language functions, we support basic 
functional programming primitiv
 * dateTime - The datetime as a long representing the milliseconds 
since unix epoch
   * Returns: The current year
 
+### `ZIP`
+  * Description: Zips lists into a single list where the ith element is an 
list containing the ith items from the constituent lists.
--- End diff --

That would great


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #652: METRON-1039: Add ZIP function to Stellar

2017-07-13 Thread cestella

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/652#discussion_r127227967
  
--- Diff: metron-stellar/stellar-common/README.md ---
@@ -711,6 +713,18 @@ In the core language functions, we support basic 
functional programming primitiv
 * dateTime - The datetime as a long representing the milliseconds 
since unix epoch
   * Returns: The current year
 
+### `ZIP`
+  * Description: Zips lists into a single list where the ith element is an 
list containing the ith items from the constituent lists.
--- End diff --

Well, there is 
https://en.wikipedia.org/wiki/Convolution_(computer_science), what do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

2017-07-13 Thread ottobackwards

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/650#discussion_r127220715
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/MathFunctions.java
 ---
@@ -25,17 +25,39 @@
 import org.apache.metron.stellar.dsl.StellarFunction;
 
 import java.util.List;
+import java.util.function.Function;
 
 public class MathFunctions {
 
-  @Stellar(name="ABS"
-  ,description="Returns the absolute value of a number."
-  ,params = {
-"number - The number to take the absolute value of"
-}
-  , returns="The absolute value of the number passed in."
-  )
-  public static class Abs implements StellarFunction {
+  private enum SingleArgMathFunctions implements Function {
--- End diff --

Why have this private? why not have stellar/common/util/math/*



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

2017-07-13 Thread ottobackwards

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/650#discussion_r127220966
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/MathFunctions.java
 ---
@@ -25,17 +25,39 @@
 import org.apache.metron.stellar.dsl.StellarFunction;
 
 import java.util.List;
+import java.util.function.Function;
 
 public class MathFunctions {
 
-  @Stellar(name="ABS"
-  ,description="Returns the absolute value of a number."
-  ,params = {
-"number - The number to take the absolute value of"
-}
-  , returns="The absolute value of the number passed in."
-  )
-  public static class Abs implements StellarFunction {
+  private enum SingleArgMathFunctions implements Function {
+ABS(d -> Math.abs(d)),
+CEIL(d -> Math.ceil(d)),
+COS(d -> Math.cos(d)),
+FLOOR(d -> Math.floor(d)),
+LOG10(d -> Math.log10(d)),
+LOG2(d -> Math.log(d)/Math.log(2)),
+LN(d -> Math.log(d)),
+SIN(d -> Math.sin(d)),
+SQRT(d -> Math.sqrt(d)),
+TAN(d -> Math.tan(d)),
+;
+
--- End diff --

Is there a reason why only this subset of Math.* is exposed?  What about 
the rest?  Are they not needed, applicable, or left for future work?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #520: METRON-833: Update MaaS documentation to explain h...

2017-07-13 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/520


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #648: METRON-1033 Corrected profiler docs units on expir...

2017-07-13 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/648


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #648: METRON-1033 Corrected profiler docs units on expires fiel...

2017-07-13 Thread nickwallen

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/648
  
+1 Thanks for the fix, Simon


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

2017-07-13 Thread iraghumitra

Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/620
  
@mraliagha Glad, it worked and thanks for persisting with the issue. 

As for supporting multiple Elasticsearch urls. Are you looking for a load 
balancer between two ES that has the same data or the two ES instances have 
different data ?.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #651: METRON-1037 Added POWER function

2017-07-13 Thread simonellistonball

GitHub user simonellistonball opened a pull request:

https://github.com/apache/metron/pull/651

METRON-1037 Added POWER function

## Contributor Comments

This is a quick addition to the Math functions. It may be worth revising 
following the work @cestella did this morning to add a more general 
implementation on top of #650 

I will probably add 2 argument capabilities and port this work to #650 when 
it's in once that is in.

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/simonellistonball/incubator-metron METRON-1037

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/651.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #651


commit 3e784245576d28ba275b1d4b3cf98347ea13f14f
Author: Simon Elliston Ball 
Date:   2017-07-13T00:51:44Z

Added POWER function




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...

2017-07-13 Thread cestella

GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/650

METRON-1038: Stellar should have a better collection of basic math 
operations

## Contributor Comments
At the moment the math functions are woefully incomplete. 
We should add at least the ones difficult or impossible to reconstruct 
using existing stellar primitives/math functions:
* log10
* log2
* ln
* sqrt
* ceil
* floor
* sin
* cos
* tan

It should be sufficient to test these out in the Stellar REPL for manual 
evaluation.  I have also refactored and added a more robust suite of unit tests 
for existing basic math functions as well as these new ones.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron METRON-1038

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/650.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #650


commit 87f1cc99144de3fd6b920c10fbe7e1c1c00d0baf
Author: cstella 
Date:   2017-07-13T09:06:10Z

METRON-1038: Stellar should have a better collection of basic math 
operations

commit ee6c609664f37c7a4fb376431a1bbcd98c08faa2
Author: cstella 
Date:   2017-07-13T09:21:38Z

Updating docs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

2017-07-13 Thread iraghumitra

Github user iraghumitra commented on the issue:

https://github.com/apache/metron/pull/620
  
@mraliagha I can get them from ES as I mentioned in my earlier comments, 
there was a bug in the code that was displaying only the fields names from bro 
indexes. It is fixed in this [commit](046c2ef4305b6317c8f0378e737f0e50af41c76a) 
. If you can check the fix on the latest code base it would be of great help.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

72 matches

Mail list logo