[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen closed the pull request at:

https://github.com/apache/metron/pull/622


---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129370546
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129356740
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,382 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(1000, 15, TimeUnit.MINUTES);
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to retrieve profile measurements 
over
+   * a time horizon.
+   *
+   * @param profile The name of the profile.
+   * @param entity The name of the entity.
+   * @param groups The group(s) used to sort the profile data.
+   * @param start When the time horizon starts in epoch milliseconds.
+   * @param end When the time horizon ends in epoch milliseconds.
+   * @return All of the row keys necessary to retrieve the profile 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129356631
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129319336
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129318110
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,382 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(1000, 15, TimeUnit.MINUTES);
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to retrieve profile measurements 
over
+   * a time horizon.
+   *
+   * @param profile The name of the profile.
+   * @param entity The name of the entity.
+   * @param groups The group(s) used to sort the profile data.
+   * @param start When the time horizon starts in epoch milliseconds.
+   * @param end When the time horizon ends in epoch milliseconds.
+   * @return All of the row keys necessary to retrieve the profile 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129309663
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/SaltyRowKeyBuilder.java
 ---
@@ -81,20 +99,19 @@ public SaltyRowKeyBuilder(int saltDivisor, long 
duration, TimeUnit units) {
* @return All of the row keys necessary to retrieve the profile 
measurements.
*/
   @Override
-  public List rowKeys(String profile, String entity, List 
groups, long start, long end) {
+  public List encode(String profile, String entity, List 
groups, long start, long end) {
 // be forgiving of out-of-order start and end times; order is critical 
to this algorithm
 end = Math.max(start, end);
 start = Math.min(start, end);
--- End diff --

This does look fishy.  I will open a separate JIRA and track this 
separately.  Thanks, @mattf-horton!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129306225
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129252984
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129252678
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129252112
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129227588
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129193892
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,382 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(1000, 15, TimeUnit.MINUTES);
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to retrieve profile measurements 
over
+   * a time horizon.
+   *
+   * @param profile The name of the profile.
+   * @param entity The name of the entity.
+   * @param groups The group(s) used to sort the profile data.
+   * @param start When the time horizon starts in epoch milliseconds.
+   * @param end When the time horizon ends in epoch milliseconds.
+   * @return All of the row keys necessary to retrieve the profile 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r129190841
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,402 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
+
+  /**
+   * A salt can be prepended to the row key to help prevent hot-spotting.  
The salt
+   * divisor is used to generate the salt.  The salt divisor should be 
roughly equal
+   * to the number of nodes in the Hbase cluster.
+   */
+  private int saltDivisor;
+
+  /**
+   * The duration of each profile period in milliseconds.
+   */
+  private long periodDurationMillis;
+
+  public DecodableRowKeyBuilder() {
+this(PROFILER_SALT_DIVISOR.getDefault(Integer.class),
+PROFILER_PERIOD.getDefault(Long.class),
+
TimeUnit.valueOf(PROFILER_PERIOD_UNITS.getDefault(String.class)));
+  }
+
+  public DecodableRowKeyBuilder(int saltDivisor, long duration, TimeUnit 
units) {
+this.saltDivisor = saltDivisor;
+this.periodDurationMillis = units.toMillis(duration);
+  }
+
+  /**
+   * Builds a list of row keys necessary to 

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r128650596
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/SaltyRowKeyBuilder.java
 ---
@@ -81,20 +99,19 @@ public SaltyRowKeyBuilder(int saltDivisor, long 
duration, TimeUnit units) {
* @return All of the row keys necessary to retrieve the profile 
measurements.
*/
   @Override
-  public List rowKeys(String profile, String entity, List 
groups, long start, long end) {
+  public List encode(String profile, String entity, List 
groups, long start, long end) {
 // be forgiving of out-of-order start and end times; order is critical 
to this algorithm
 end = Math.max(start, end);
 start = Math.min(start, end);
--- End diff --

Heh, this has been in the code for a long time, but isn't this a bug?  If 
it starts out in the wrong order, say end is 1 and start is 5, won't this pair 
of statements result in both end and start being equal to the larger, ie 5 ?  
We need an intermediate variable for a binary swap!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127331446
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/RowKeyBuilderFactory.java
 ---
@@ -0,0 +1,125 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.beanutils.PropertyUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.InvocationTargetException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_ROW_KEY_BUILDER;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * A Factory class that can create a RowKeyBuilder based on global 
property values.
+ */
+public class RowKeyBuilderFactory {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(RowKeyBuilderFactory.class);
+
+  /**
+   * Create a RowKeyBuilder.
+   * @param global The global properties.
+   * @return A RowKeyBuilder instantiated using the global property values.
+   */
+  public static RowKeyBuilder create(Map global) {
+String rowKeyBuilderClass = PROFILER_ROW_KEY_BUILDER.get(global, 
String.class);
+LOG.debug("profiler client: {}={}", PROFILER_ROW_KEY_BUILDER, 
rowKeyBuilderClass);
+
+// instantiate the RowKeyBuilder
+RowKeyBuilder builder = 
ReflectionUtils.createInstance(rowKeyBuilderClass);
+setSaltDivisor(global, builder);
+setPeriodDuration(global, builder);
--- End diff --

If I had some IoC-like functionality like Flux or Spring here, then this 
wouldn't be a problem at all.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127326080
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/RowKeyBuilderFactory.java
 ---
@@ -0,0 +1,125 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.beanutils.PropertyUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.InvocationTargetException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_ROW_KEY_BUILDER;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * A Factory class that can create a RowKeyBuilder based on global 
property values.
+ */
+public class RowKeyBuilderFactory {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(RowKeyBuilderFactory.class);
+
+  /**
+   * Create a RowKeyBuilder.
+   * @param global The global properties.
+   * @return A RowKeyBuilder instantiated using the global property values.
+   */
+  public static RowKeyBuilder create(Map global) {
+String rowKeyBuilderClass = PROFILER_ROW_KEY_BUILDER.get(global, 
String.class);
+LOG.debug("profiler client: {}={}", PROFILER_ROW_KEY_BUILDER, 
rowKeyBuilderClass);
+
+// instantiate the RowKeyBuilder
+RowKeyBuilder builder = 
ReflectionUtils.createInstance(rowKeyBuilderClass);
+setSaltDivisor(global, builder);
+setPeriodDuration(global, builder);
--- End diff --

I don't really like how I go about setting the salt divisor and period 
duration on the `RowKeyBuilder`.  There are no methods in the `RowKeyBuilder` 
interface to do set these values.  I could add something like 
`RowKeyBuilder.setSaltDivisor`, but I was trying not to pollute that interface 
with variables like salt divisor that may not apply to all RowKeyBuilder 
implementations.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127329675
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/SaltyRowKeyBuilder.java
 ---
@@ -44,7 +46,17 @@
  * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
  * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
  * 
+ *
+ * This row key builder has no logic to decode a row key, nor is the row 
key generated by this builder
+ * easily decodable.  More specifically, the profile, entity, groups and 
period that make up the row key
+ * cannot be extracted from a previously generated row key.  This makes it 
difficult to answer questions
+ * like; What entities are included in this profile?  What is the period 
for this profile?  Use the
+ * DecodableRowKeyBuilder instead.
+ *
+ * @deprecated Replaced by DecodableRowKeyBuilder
+ * @see DecodableRowKeyBuilder
  */
+@Deprecated
 public class SaltyRowKeyBuilder implements RowKeyBuilder {
--- End diff --

I marked the old `RowKeyBuilder` as deprecated.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127326600
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/RowKeyBuilderFactory.java
 ---
@@ -0,0 +1,125 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.beanutils.PropertyUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.InvocationTargetException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_ROW_KEY_BUILDER;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * A Factory class that can create a RowKeyBuilder based on global 
property values.
+ */
+public class RowKeyBuilderFactory {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(RowKeyBuilderFactory.class);
+
+  /**
+   * Create a RowKeyBuilder.
+   * @param global The global properties.
+   * @return A RowKeyBuilder instantiated using the global property values.
+   */
+  public static RowKeyBuilder create(Map global) {
+String rowKeyBuilderClass = PROFILER_ROW_KEY_BUILDER.get(global, 
String.class);
+LOG.debug("profiler client: {}={}", PROFILER_ROW_KEY_BUILDER, 
rowKeyBuilderClass);
+
+// instantiate the RowKeyBuilder
+RowKeyBuilder builder = 
ReflectionUtils.createInstance(rowKeyBuilderClass);
+setSaltDivisor(global, builder);
+setPeriodDuration(global, builder);
--- End diff --

But I think this actually turned out worse, than the alternative of just 
adding `RowKeyBuilder.setSaltDivisor` and polluting the interface.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127325245
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/RowKeyBuilderFactory.java
 ---
@@ -0,0 +1,125 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.beanutils.PropertyUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.InvocationTargetException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_ROW_KEY_BUILDER;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * A Factory class that can create a RowKeyBuilder based on global 
property values.
+ */
+public class RowKeyBuilderFactory {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(RowKeyBuilderFactory.class);
+
+  /**
+   * Create a RowKeyBuilder.
+   * @param global The global properties.
+   * @return A RowKeyBuilder instantiated using the global property values.
+   */
+  public static RowKeyBuilder create(Map global) {
+String rowKeyBuilderClass = PROFILER_ROW_KEY_BUILDER.get(global, 
String.class);
+LOG.debug("profiler client: {}={}", PROFILER_ROW_KEY_BUILDER, 
rowKeyBuilderClass);
+
+// instantiate the RowKeyBuilder
+RowKeyBuilder builder = 
ReflectionUtils.createInstance(rowKeyBuilderClass);
+setSaltDivisor(global, builder);
+setPeriodDuration(global, builder);
--- End diff --

Here is the logic to instantiate a `RowKeyBuilder` that is used by the 
Profiler Client's `GetProfile`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127330773
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,382 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
+
+  /**
+   * Defines the byte order when encoding and decoding the row keys.
+   *
+   * Making this configurable is likely not necessary and is left as a 
practice exercise for the reader. :)
+   */
+  private static final ByteOrder byteOrder = ByteOrder.BIG_ENDIAN;
+
+  /**
+   * Defines some level of sane max field length to avoid any shenanigans 
with oddly encoded row keys.
+   */
+  private static final int MAX_FIELD_LENGTH = 1000;
+
+  /**
+   * A magic number embedded in each row key to help validate the row key 
and byte ordering when decoding.
+   */
+  protected static final short MAGIC_NUMBER = 77;
+
+  /**
+   * The version number of the row keys supported by this builder.
+   */
+  protected static final byte VERSION = (byte) 1;
--- End diff --

I added a `VERSION` field to the row key, hoping that this might help 
future changes to the `RowKeyBuilder`.  With this, I could potentially start to 
parse the row key and then choose the right `RowKeyBuilder` implementation; the 
one used to create the row key.  This would make row key changes seemless to 
users.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127329289
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/RowKeyBuilderFactory.java
 ---
@@ -0,0 +1,125 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client.stellar;
+
+import org.apache.commons.beanutils.PropertyUtils;
+import org.apache.commons.lang3.ClassUtils;
+import org.apache.metron.common.utils.ReflectionUtils;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.reflect.InvocationTargetException;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_PERIOD_UNITS;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_ROW_KEY_BUILDER;
+import static 
org.apache.metron.profiler.client.stellar.ProfilerConfig.PROFILER_SALT_DIVISOR;
+
+/**
+ * A Factory class that can create a RowKeyBuilder based on global 
property values.
+ */
+public class RowKeyBuilderFactory {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(RowKeyBuilderFactory.class);
+
+  /**
+   * Create a RowKeyBuilder.
+   * @param global The global properties.
+   * @return A RowKeyBuilder instantiated using the global property values.
+   */
+  public static RowKeyBuilder create(Map global) {
+String rowKeyBuilderClass = PROFILER_ROW_KEY_BUILDER.get(global, 
String.class);
+LOG.debug("profiler client: {}={}", PROFILER_ROW_KEY_BUILDER, 
rowKeyBuilderClass);
+
+// instantiate the RowKeyBuilder
+RowKeyBuilder builder = 
ReflectionUtils.createInstance(rowKeyBuilderClass);
+setSaltDivisor(global, builder);
+setPeriodDuration(global, builder);
+
+return builder;
+  }
+
+  /**
+   * Set the period duration on the RowKeyBuilder.
+   * @param global The global properties from Zk.
+   * @param builder The RowKeyBuilder implementation.
+   */
+  private static void setPeriodDuration(Map global, 
RowKeyBuilder builder) {
+
+// how long is the profile period?
+long duration = PROFILER_PERIOD.get(global, Long.class);
+LOG.debug("profiler client: {}={}", PROFILER_PERIOD, duration);
+
+// which units are used to define the profile period?
+String configuredUnits = PROFILER_PERIOD_UNITS.get(global, 
String.class);
+TimeUnit units = TimeUnit.valueOf(configuredUnits);
+LOG.debug("profiler client: {}={}", PROFILER_PERIOD_UNITS, units);
+
+// set the period duration
+final String periodDurationProperty = "periodDurationMillis";
+setProperty(builder, periodDurationProperty, units.toMillis(duration));
+  }
+
+  /**
+   * Set the salt divisor property on the RowKeyBuilder.
+   * @param global The global properties from Zk.
+   * @param builder The RowKeyBuilder implementation.
+   */
+  private static void setSaltDivisor(Map global, 
RowKeyBuilder builder) {
+
+// what is the salt divisor?
+Integer saltDivisor = PROFILER_SALT_DIVISOR.get(global, Integer.class);
+LOG.debug("profiler client: {}={}", PROFILER_SALT_DIVISOR, 
saltDivisor);
+
+final String saltDivisorProperty = "saltDivisor";
+setProperty(builder, saltDivisorProperty, saltDivisor);
--- End diff --

This basically sets the 'salt divisor' on any `RowKeyBuilder` that has a 
`saltDivisor` setter.  I really don't like this.  It is very hack-ish. I would 
love to use a simpler alternative.


---
If your project is set up for it, you can reply to this email and have your

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127328660
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java
 ---
@@ -216,21 +211,7 @@ private ColumnBuilder getColumnBuilder(Map global) {
* @param global The global configuration.
*/
   private RowKeyBuilder getRowKeyBuilder(Map global) {
-
-// how long is the profile period?
-long duration = PROFILER_PERIOD.get(global, Long.class);
-LOG.debug("profiler client: {}={}", PROFILER_PERIOD, duration);
-
-// which units are used to define the profile period?
-String configuredUnits = PROFILER_PERIOD_UNITS.get(global, 
String.class);
-TimeUnit units = TimeUnit.valueOf(configuredUnits);
-LOG.debug("profiler client: {}={}", PROFILER_PERIOD_UNITS, units);
-
-// what is the salt divisor?
-Integer saltDivisor = PROFILER_SALT_DIVISOR.get(global, Integer.class);
-LOG.debug("profiler client: {}={}", PROFILER_SALT_DIVISOR, 
saltDivisor);
-
-return new SaltyRowKeyBuilder(saltDivisor, duration, units);
+return RowKeyBuilderFactory.create(global);
--- End diff --

This is where we need to instantiate the `RowKeyBuilder` for the Profiler 
Client API.  Like I will discuss in another thread, the logic got complex and 
kind of nasty so I encapsulated it in its own `RowKeyBuilderFactory`.  See that 
class for a further discussion as to why it is kind of nasty.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127329548
  
--- Diff: 
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/DecodableRowKeyBuilder.java
 ---
@@ -0,0 +1,382 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.hbase;
+
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+
+import java.nio.BufferUnderflowException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Responsible for building the row keys used to store profile data in 
HBase.
+ *
+ * This builder generates decodable row keys.  A decodable row key is one 
that can be interrogated to extract
+ * the constituent components of that row key.  Given a previously 
generated row key this builder
+ * can extract the profile name, entity name, group name(s), period 
duration, and period.
+ *
+ * The row key is composed of the following fields.
+ * 
+ * magic number - Helps to validate the row key.
+ * version - The version number of the row key.
+ * salt - A salt that helps prevent hot-spotting.
+ * profile - The name of the profile.
+ * entity - The name of the entity being profiled.
+ * group(s) - The group(s) used to sort the data in HBase. For 
example, a group may distinguish between weekends and weekdays.
+ * period - The period in which the measurement was taken. The first 
period starts at the epoch and increases monotonically.
+ * 
+ */
+public class DecodableRowKeyBuilder implements RowKeyBuilder {
--- End diff --

The new `RowKeyBuilder` implementation that is decodable.  Everyone should 
just use this, but the older implementation is left for backwards compatibility.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/622#discussion_r127327126
  
--- Diff: 
metron-analytics/metron-profiler/src/main/flux/profiler/remote.yaml ---
@@ -29,7 +29,7 @@ components:
 - name: "saltDivisor"
--- End diff --

Notice that the legacy `RowKeyBuilder`, the `SaltyRowKeyBuilder`, is still 
the default.  If a user wants to use the new `RowKeyBuilder` then they need to 
change the flux file here and specify 
`org.apache.metron.profiler.hbase.DecodableRowKeyBuilder`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #622: METRON-1005 Create Decodable Row Key for Profiler

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/622

METRON-1005 Create Decodable Row Key for Profiler

To be able to answer the types of questions that I outlined in 
[METRON-450](https://issues.apache.org/jira/browse/METRON-450), we need a row 
key that is decodable. Right now there is no logic to decode a row key, nor is 
the existing row key easily decodable.

Once the row keys can be decoded, you could scan all of the row keys in the 
Profiler's HBase table, decode each of them and extract things like, the names 
of all your profiles, the names of entities within a profile, the period 
duration of a given profile.

- [ ] Do not merge.  Opening this PR for review and feedback.  I still need 
to run this through manual testing.

- [ ] **WARNING**: This change is NOT backwards compatible.  The row key 
format has changed.  All data written with the legacy row key cannot be read by 
the new code in this PR.  Should I make this backwards compatible?  I could 
make these changes live in a new `RowKeyBuilder` implementation, which would 
allow for backwards compatibility.

## Pull Request Checklist
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?





You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1005

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/622.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #622


commit edd7fa6946b4bcb5975c8497ab5550201718e426
Author: Nick Allen 
Date:   2017-06-22T21:45:40Z

METRON-1005 Create Decodable Row Key for Profiler




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---