[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2 URL: https://github.com/apache/metron/pull/1458#discussion_r312157858 ## File path: dependencies_with_url.csv ## @@ -146,6 +146,7 @@ com.codahale.metrics:metrics-core:jar:3.0.2:compile,MIT,https://github.com/codah com.codahale.metrics:metrics-graphite:jar:3.0.2:compile,MIT,https://github.com/codahale/metrics com.esotericsoftware.reflectasm:reflectasm:jar:shaded:1.07:compile,BSD,https://github.com/EsotericSoftware/reflectasm com.fasterxml.jackson.core:jackson-annotations:jar:2.2.3:compile,ASLv2,http://wiki.fasterxml.com/JacksonHome +com.fasterxml.jackson.core:jackson-annotations:jar:2.7.0:compile,ASLv2,http://wiki.fasterxml.com/JacksonHome Review comment: See 1ef49cfec. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2 URL: https://github.com/apache/metron/pull/1458#discussion_r312156007 ## File path: dependencies_with_url.csv ## @@ -146,6 +146,7 @@ com.codahale.metrics:metrics-core:jar:3.0.2:compile,MIT,https://github.com/codah com.codahale.metrics:metrics-graphite:jar:3.0.2:compile,MIT,https://github.com/codahale/metrics com.esotericsoftware.reflectasm:reflectasm:jar:shaded:1.07:compile,BSD,https://github.com/EsotericSoftware/reflectasm com.fasterxml.jackson.core:jackson-annotations:jar:2.2.3:compile,ASLv2,http://wiki.fasterxml.com/JacksonHome +com.fasterxml.jackson.core:jackson-annotations:jar:2.7.0:compile,ASLv2,http://wiki.fasterxml.com/JacksonHome Review comment: Actually, I think I can also explicitly pull-in `jackson-annotations:2.7.4` in the pom to avoid pulling in `jackson-annotations:2.7.0` as a transitive. That is what the other projects seem to do like `metron-parsers-common`. That will avoid pulling in some other random version like 2.7.0. I have no idea why a transitive for jackson-databind:2.7.4 would be jackson-annotations:2.7.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311720802
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
##
@@ -20,437 +20,182 @@
package org.apache.metron.profiler.client.stellar;
-import org.apache.hadoop.hbase.client.HTableInterface;
-import org.apache.metron.hbase.mock.MockHBaseTableProvider;
import org.apache.metron.profiler.ProfileMeasurement;
-import org.apache.metron.profiler.client.ProfileWriter;
-import org.apache.metron.profiler.hbase.ColumnBuilder;
-import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
-import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+import org.apache.metron.profiler.client.ProfilerClient;
import org.apache.metron.stellar.common.DefaultStellarStatefulExecutor;
import org.apache.metron.stellar.common.StellarStatefulExecutor;
import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.functions.resolver.FunctionResolver;
import org.apache.metron.stellar.dsl.functions.resolver.SimpleFunctionResolver;
-import
org.apache.metron.stellar.dsl.functions.resolver.SingletonFunctionResolver;
-import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.TimeUnit;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_COLUMN_FAMILY;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE_PROVIDER;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
/**
- * Tests the GetProfile class.
+ * Tests the 'PROFILE_GET' function in the {@link GetProfile} class.
*/
public class GetProfileTest {
- private static final long periodDuration = 15;
- private static final TimeUnit periodUnits = TimeUnit.MINUTES;
- private static final int saltDivisor = 1000;
- private static final String tableName = "profiler";
- private static final String columnFamily = "P";
private StellarStatefulExecutor executor;
- private Map state;
- private ProfileWriter profileWriter;
- // different values of period and salt divisor, used to test
config_overrides feature
- private static final long periodDuration2 = 1;
- private static final TimeUnit periodUnits2 = TimeUnit.HOURS;
- private static final int saltDivisor2 = 2050;
-
- private T run(String expression, Class clazz) {
-return executor.execute(expression, state, clazz);
+ private FunctionResolver functionResolver;
+ private Map globals;
+ private GetProfile function;
+ private ProfilerClient profilerClient;
+ private List results;
+ private ProfileMeasurement expected;
+
+ private List run(String expression) {
+return executor.execute(expression, new HashMap<>(), List.class);
}
- /**
- * This method sets up the configuration context for both writing profile
data
- * (using profileWriter to mock the complex process of what the Profiler
topology
- * actually does), and then reading that profile data (thereby testing the
PROFILE_GET
- * Stellar client implemented in GetProfile).
- *
- * It runs at @Before time, and sets testclass global variables used by the
writers and readers.
- * The various writers and readers are in each test case, not here.
- *
- * @return void
- */
@Before
public void setup() {
-state = new HashMap<>();
-final HTableInterface table = MockHBaseTableProvider.addToCache(tableName,
columnFamily);
-
-// used to write values to be read during testing
-long periodDurationMillis = TimeUnit.MINUTES.toMillis(15);
-RowKeyBuilder rowKeyBuilder = new SaltyRowKeyBuilder();
-ColumnBuilder columnBuilder = new ValueOnlyColumnBuilder(columnFamily);
-profileWriter = new ProfileWriter(rowKeyBuilder, columnBuilder, table,
periodDurationMillis);
-
-// global properties
-Map global = new HashMap() {{
- put(PROFILER_HBASE_TABLE.getKey(), tableName);
- put(PROFILER_COLUMN_FAMILY.getKey(), columnFamily);
-
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311720802
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
##
@@ -20,437 +20,182 @@
package org.apache.metron.profiler.client.stellar;
-import org.apache.hadoop.hbase.client.HTableInterface;
-import org.apache.metron.hbase.mock.MockHBaseTableProvider;
import org.apache.metron.profiler.ProfileMeasurement;
-import org.apache.metron.profiler.client.ProfileWriter;
-import org.apache.metron.profiler.hbase.ColumnBuilder;
-import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
-import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+import org.apache.metron.profiler.client.ProfilerClient;
import org.apache.metron.stellar.common.DefaultStellarStatefulExecutor;
import org.apache.metron.stellar.common.StellarStatefulExecutor;
import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.functions.resolver.FunctionResolver;
import org.apache.metron.stellar.dsl.functions.resolver.SimpleFunctionResolver;
-import
org.apache.metron.stellar.dsl.functions.resolver.SingletonFunctionResolver;
-import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.TimeUnit;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_COLUMN_FAMILY;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE_PROVIDER;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
/**
- * Tests the GetProfile class.
+ * Tests the 'PROFILE_GET' function in the {@link GetProfile} class.
*/
public class GetProfileTest {
- private static final long periodDuration = 15;
- private static final TimeUnit periodUnits = TimeUnit.MINUTES;
- private static final int saltDivisor = 1000;
- private static final String tableName = "profiler";
- private static final String columnFamily = "P";
private StellarStatefulExecutor executor;
- private Map state;
- private ProfileWriter profileWriter;
- // different values of period and salt divisor, used to test
config_overrides feature
- private static final long periodDuration2 = 1;
- private static final TimeUnit periodUnits2 = TimeUnit.HOURS;
- private static final int saltDivisor2 = 2050;
-
- private T run(String expression, Class clazz) {
-return executor.execute(expression, state, clazz);
+ private FunctionResolver functionResolver;
+ private Map globals;
+ private GetProfile function;
+ private ProfilerClient profilerClient;
+ private List results;
+ private ProfileMeasurement expected;
+
+ private List run(String expression) {
+return executor.execute(expression, new HashMap<>(), List.class);
}
- /**
- * This method sets up the configuration context for both writing profile
data
- * (using profileWriter to mock the complex process of what the Profiler
topology
- * actually does), and then reading that profile data (thereby testing the
PROFILE_GET
- * Stellar client implemented in GetProfile).
- *
- * It runs at @Before time, and sets testclass global variables used by the
writers and readers.
- * The various writers and readers are in each test case, not here.
- *
- * @return void
- */
@Before
public void setup() {
-state = new HashMap<>();
-final HTableInterface table = MockHBaseTableProvider.addToCache(tableName,
columnFamily);
-
-// used to write values to be read during testing
-long periodDurationMillis = TimeUnit.MINUTES.toMillis(15);
-RowKeyBuilder rowKeyBuilder = new SaltyRowKeyBuilder();
-ColumnBuilder columnBuilder = new ValueOnlyColumnBuilder(columnFamily);
-profileWriter = new ProfileWriter(rowKeyBuilder, columnBuilder, table,
periodDurationMillis);
-
-// global properties
-Map global = new HashMap() {{
- put(PROFILER_HBASE_TABLE.getKey(), tableName);
- put(PROFILER_COLUMN_FAMILY.getKey(), columnFamily);
-
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311720802
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
##
@@ -20,437 +20,182 @@
package org.apache.metron.profiler.client.stellar;
-import org.apache.hadoop.hbase.client.HTableInterface;
-import org.apache.metron.hbase.mock.MockHBaseTableProvider;
import org.apache.metron.profiler.ProfileMeasurement;
-import org.apache.metron.profiler.client.ProfileWriter;
-import org.apache.metron.profiler.hbase.ColumnBuilder;
-import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
-import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+import org.apache.metron.profiler.client.ProfilerClient;
import org.apache.metron.stellar.common.DefaultStellarStatefulExecutor;
import org.apache.metron.stellar.common.StellarStatefulExecutor;
import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.functions.resolver.FunctionResolver;
import org.apache.metron.stellar.dsl.functions.resolver.SimpleFunctionResolver;
-import
org.apache.metron.stellar.dsl.functions.resolver.SingletonFunctionResolver;
-import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.TimeUnit;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_COLUMN_FAMILY;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE_PROVIDER;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
/**
- * Tests the GetProfile class.
+ * Tests the 'PROFILE_GET' function in the {@link GetProfile} class.
*/
public class GetProfileTest {
- private static final long periodDuration = 15;
- private static final TimeUnit periodUnits = TimeUnit.MINUTES;
- private static final int saltDivisor = 1000;
- private static final String tableName = "profiler";
- private static final String columnFamily = "P";
private StellarStatefulExecutor executor;
- private Map state;
- private ProfileWriter profileWriter;
- // different values of period and salt divisor, used to test
config_overrides feature
- private static final long periodDuration2 = 1;
- private static final TimeUnit periodUnits2 = TimeUnit.HOURS;
- private static final int saltDivisor2 = 2050;
-
- private T run(String expression, Class clazz) {
-return executor.execute(expression, state, clazz);
+ private FunctionResolver functionResolver;
+ private Map globals;
+ private GetProfile function;
+ private ProfilerClient profilerClient;
+ private List results;
+ private ProfileMeasurement expected;
+
+ private List run(String expression) {
+return executor.execute(expression, new HashMap<>(), List.class);
}
- /**
- * This method sets up the configuration context for both writing profile
data
- * (using profileWriter to mock the complex process of what the Profiler
topology
- * actually does), and then reading that profile data (thereby testing the
PROFILE_GET
- * Stellar client implemented in GetProfile).
- *
- * It runs at @Before time, and sets testclass global variables used by the
writers and readers.
- * The various writers and readers are in each test case, not here.
- *
- * @return void
- */
@Before
public void setup() {
-state = new HashMap<>();
-final HTableInterface table = MockHBaseTableProvider.addToCache(tableName,
columnFamily);
-
-// used to write values to be read during testing
-long periodDurationMillis = TimeUnit.MINUTES.toMillis(15);
-RowKeyBuilder rowKeyBuilder = new SaltyRowKeyBuilder();
-ColumnBuilder columnBuilder = new ValueOnlyColumnBuilder(columnFamily);
-profileWriter = new ProfileWriter(rowKeyBuilder, columnBuilder, table,
periodDurationMillis);
-
-// global properties
-Map global = new HashMap() {{
- put(PROFILER_HBASE_TABLE.getKey(), tableName);
- put(PROFILER_COLUMN_FAMILY.getKey(), columnFamily);
-
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311720802
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
##
@@ -20,437 +20,182 @@
package org.apache.metron.profiler.client.stellar;
-import org.apache.hadoop.hbase.client.HTableInterface;
-import org.apache.metron.hbase.mock.MockHBaseTableProvider;
import org.apache.metron.profiler.ProfileMeasurement;
-import org.apache.metron.profiler.client.ProfileWriter;
-import org.apache.metron.profiler.hbase.ColumnBuilder;
-import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
-import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+import org.apache.metron.profiler.client.ProfilerClient;
import org.apache.metron.stellar.common.DefaultStellarStatefulExecutor;
import org.apache.metron.stellar.common.StellarStatefulExecutor;
import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.functions.resolver.FunctionResolver;
import org.apache.metron.stellar.dsl.functions.resolver.SimpleFunctionResolver;
-import
org.apache.metron.stellar.dsl.functions.resolver.SingletonFunctionResolver;
-import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.TimeUnit;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_COLUMN_FAMILY;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE_PROVIDER;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
/**
- * Tests the GetProfile class.
+ * Tests the 'PROFILE_GET' function in the {@link GetProfile} class.
*/
public class GetProfileTest {
- private static final long periodDuration = 15;
- private static final TimeUnit periodUnits = TimeUnit.MINUTES;
- private static final int saltDivisor = 1000;
- private static final String tableName = "profiler";
- private static final String columnFamily = "P";
private StellarStatefulExecutor executor;
- private Map state;
- private ProfileWriter profileWriter;
- // different values of period and salt divisor, used to test
config_overrides feature
- private static final long periodDuration2 = 1;
- private static final TimeUnit periodUnits2 = TimeUnit.HOURS;
- private static final int saltDivisor2 = 2050;
-
- private T run(String expression, Class clazz) {
-return executor.execute(expression, state, clazz);
+ private FunctionResolver functionResolver;
+ private Map globals;
+ private GetProfile function;
+ private ProfilerClient profilerClient;
+ private List results;
+ private ProfileMeasurement expected;
+
+ private List run(String expression) {
+return executor.execute(expression, new HashMap<>(), List.class);
}
- /**
- * This method sets up the configuration context for both writing profile
data
- * (using profileWriter to mock the complex process of what the Profiler
topology
- * actually does), and then reading that profile data (thereby testing the
PROFILE_GET
- * Stellar client implemented in GetProfile).
- *
- * It runs at @Before time, and sets testclass global variables used by the
writers and readers.
- * The various writers and readers are in each test case, not here.
- *
- * @return void
- */
@Before
public void setup() {
-state = new HashMap<>();
-final HTableInterface table = MockHBaseTableProvider.addToCache(tableName,
columnFamily);
-
-// used to write values to be read during testing
-long periodDurationMillis = TimeUnit.MINUTES.toMillis(15);
-RowKeyBuilder rowKeyBuilder = new SaltyRowKeyBuilder();
-ColumnBuilder columnBuilder = new ValueOnlyColumnBuilder(columnFamily);
-profileWriter = new ProfileWriter(rowKeyBuilder, columnBuilder, table,
periodDurationMillis);
-
-// global properties
-Map global = new HashMap() {{
- put(PROFILER_HBASE_TABLE.getKey(), tableName);
- put(PROFILER_COLUMN_FAMILY.getKey(), columnFamily);
-
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311720802
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
##
@@ -20,437 +20,182 @@
package org.apache.metron.profiler.client.stellar;
-import org.apache.hadoop.hbase.client.HTableInterface;
-import org.apache.metron.hbase.mock.MockHBaseTableProvider;
import org.apache.metron.profiler.ProfileMeasurement;
-import org.apache.metron.profiler.client.ProfileWriter;
-import org.apache.metron.profiler.hbase.ColumnBuilder;
-import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
-import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+import org.apache.metron.profiler.client.ProfilerClient;
import org.apache.metron.stellar.common.DefaultStellarStatefulExecutor;
import org.apache.metron.stellar.common.StellarStatefulExecutor;
import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.functions.resolver.FunctionResolver;
import org.apache.metron.stellar.dsl.functions.resolver.SimpleFunctionResolver;
-import
org.apache.metron.stellar.dsl.functions.resolver.SingletonFunctionResolver;
-import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.TimeUnit;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_COLUMN_FAMILY;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE_PROVIDER;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
/**
- * Tests the GetProfile class.
+ * Tests the 'PROFILE_GET' function in the {@link GetProfile} class.
*/
public class GetProfileTest {
- private static final long periodDuration = 15;
- private static final TimeUnit periodUnits = TimeUnit.MINUTES;
- private static final int saltDivisor = 1000;
- private static final String tableName = "profiler";
- private static final String columnFamily = "P";
private StellarStatefulExecutor executor;
- private Map state;
- private ProfileWriter profileWriter;
- // different values of period and salt divisor, used to test
config_overrides feature
- private static final long periodDuration2 = 1;
- private static final TimeUnit periodUnits2 = TimeUnit.HOURS;
- private static final int saltDivisor2 = 2050;
-
- private T run(String expression, Class clazz) {
-return executor.execute(expression, state, clazz);
+ private FunctionResolver functionResolver;
+ private Map globals;
+ private GetProfile function;
+ private ProfilerClient profilerClient;
+ private List results;
+ private ProfileMeasurement expected;
+
+ private List run(String expression) {
+return executor.execute(expression, new HashMap<>(), List.class);
}
- /**
- * This method sets up the configuration context for both writing profile
data
- * (using profileWriter to mock the complex process of what the Profiler
topology
- * actually does), and then reading that profile data (thereby testing the
PROFILE_GET
- * Stellar client implemented in GetProfile).
- *
- * It runs at @Before time, and sets testclass global variables used by the
writers and readers.
- * The various writers and readers are in each test case, not here.
- *
- * @return void
- */
@Before
public void setup() {
-state = new HashMap<>();
-final HTableInterface table = MockHBaseTableProvider.addToCache(tableName,
columnFamily);
-
-// used to write values to be read during testing
-long periodDurationMillis = TimeUnit.MINUTES.toMillis(15);
-RowKeyBuilder rowKeyBuilder = new SaltyRowKeyBuilder();
-ColumnBuilder columnBuilder = new ValueOnlyColumnBuilder(columnFamily);
-profileWriter = new ProfileWriter(rowKeyBuilder, columnBuilder, table,
periodDurationMillis);
-
-// global properties
-Map global = new HashMap() {{
- put(PROFILER_HBASE_TABLE.getKey(), tableName);
- put(PROFILER_COLUMN_FAMILY.getKey(), columnFamily);
-
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311277636
##
File path: dependencies_with_url.csv
##
@@ -146,6 +146,7 @@
com.codahale.metrics:metrics-core:jar:3.0.2:compile,MIT,https://github.com/codah
com.codahale.metrics:metrics-graphite:jar:3.0.2:compile,MIT,https://github.com/codahale/metrics
com.esotericsoftware.reflectasm:reflectasm:jar:shaded:1.07:compile,BSD,https://github.com/EsotericSoftware/reflectasm
com.fasterxml.jackson.core:jackson-annotations:jar:2.2.3:compile,ASLv2,http://wiki.fasterxml.com/JacksonHome
+com.fasterxml.jackson.core:jackson-annotations:jar:2.7.0:compile,ASLv2,http://wiki.fasterxml.com/JacksonHome
Review comment:
Actually, jackson-databind:2.7.4, which is the version defined by
`${global_jackson_version}` pulls in jackson-annotations:2.7.0, so this seems
to be a necessary dependency.
The only remotely related changes is that I had to add the
`jackson-databind` dependency to `metron-profiler-storm`. But I am just pulling
it in explicitly, just like many other projects already do;
`metron-parsers-common`, `metron-enrichment-storm`, `stellar-common`.
I am not sure why the license validator did not pick-up on this dependency
before, since the same dependency lives in many other projects. Hmm.
```
@ ~/Development/metron/metron-analytics/metron-profiler-storm
$ mvn dependency:tree
[INFO] Scanning for projects...
[INFO]
[INFO]
[INFO] Building metron-profiler-storm 0.7.2
[INFO]
[INFO]
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @
metron-profiler-storm ---
[INFO] org.apache.metron:metron-profiler-storm:jar:0.7.2
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.7.4:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.7.0:compile
[INFO] | \- com.fasterxml.jackson.core:jackson-core:jar:2.7.4:compile
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2 URL: https://github.com/apache/metron/pull/1458#discussion_r311277636 ## File path: dependencies_with_url.csv ## @@ -146,6 +146,7 @@ com.codahale.metrics:metrics-core:jar:3.0.2:compile,MIT,https://github.com/codah com.codahale.metrics:metrics-graphite:jar:3.0.2:compile,MIT,https://github.com/codahale/metrics com.esotericsoftware.reflectasm:reflectasm:jar:shaded:1.07:compile,BSD,https://github.com/EsotericSoftware/reflectasm com.fasterxml.jackson.core:jackson-annotations:jar:2.2.3:compile,ASLv2,http://wiki.fasterxml.com/JacksonHome +com.fasterxml.jackson.core:jackson-annotations:jar:2.7.0:compile,ASLv2,http://wiki.fasterxml.com/JacksonHome Review comment: Actually, jackson-databind:2.7.4, which is the version defined by $global_jackson_version pulls in jackson-annotations:2.7.0, so this seems to be necessary. I had to move the jackson-databind dependency up so as to not conflict. I am not sure why the license validator did not pick-up on this before. ``` @ ~/Development/metron/metron-analytics/metron-profiler-storm $ mvn dependency:tree [INFO] Scanning for projects... [INFO] [INFO] [INFO] Building metron-profiler-storm 0.7.2 [INFO] [INFO] [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ metron-profiler-storm --- [INFO] org.apache.metron:metron-profiler-storm:jar:0.7.2 [INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.7.4:compile [INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.7.0:compile [INFO] | \- com.fasterxml.jackson.core:jackson-core:jar:2.7.4:compile This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311274921
##
File path:
metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/HBaseWriterFunction.java
##
@@ -137,35 +150,38 @@ public HBaseWriterFunction(Properties properties) {
}
/**
- * Set the {@link TableProvider} using the class name of the provider.
- * @param providerImpl The name of the class.
- * @return
+ * Creates an {@link HBaseConnectionFactory} based on a class name.
+ * @param factoryImpl The class name of an {@link HBaseConnectionFactory}
implementation.
*/
- public HBaseWriterFunction withTableProviderImpl(String providerImpl) {
-this.tableProvider = createTableProvider(providerImpl);
-return this;
- }
-
- /**
- * Creates a TableProvider based on a class name.
- * @param providerImpl The class name of a TableProvider
- */
- private static TableProvider createTableProvider(String providerImpl) {
-LOG.trace("Creating table provider; className={}", providerImpl);
+ private static HBaseConnectionFactory createConnectionFactory(String
factoryImpl) {
+LOG.trace("Creating table provider; className={}", factoryImpl);
// if class name not defined, use a reasonable default
-if(StringUtils.isEmpty(providerImpl) || providerImpl.charAt(0) == '$') {
- return new HTableProvider();
+if(StringUtils.isEmpty(factoryImpl) || factoryImpl.charAt(0) == '$') {
+ return new HBaseConnectionFactory();
}
// instantiate the table provider
-try {
- Class clazz = (Class)
Class.forName(providerImpl);
- return clazz.getConstructor().newInstance();
+return HBaseConnectionFactory.byName(factoryImpl);
+ }
-} catch (InstantiationException | IllegalAccessException |
IllegalStateException |
-InvocationTargetException | NoSuchMethodException |
ClassNotFoundException e) {
- throw new IllegalStateException("Unable to instantiate connector", e);
-}
+ protected HBaseWriterFunction withConnectionFactory(HBaseConnectionFactory
connectionFactory) {
Review comment:
> What happens if you pass in nulls or forget to call all the builder
methods?
I provided sensible defaults now. See 67fe5ab56.
> Which of those builder methods is absolutely required and, as a consumer
of the API...
None are absolutely required thanks to the defaults.
> how do I know which builder method wins where there is overlap in the
underlying options?
Whichever you call last, which seems intuitive to me. In reality,
withProperties() is used in production (see BatchProfiler), whereas the other
methods are used during testing.
The alternative, is to not have `Builder.withProperties` and move that
logic back as a constructor; `HBaseWriterFunction(Properties)`. When I created
the Builder, it seemed logical to move that to the Builder, but maybe you
disagree.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311266173
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/HBaseProfileWriter.java
##
@@ -0,0 +1,63 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.profiler.client;
+
+import org.apache.hadoop.hbase.client.Durability;
+import org.apache.metron.hbase.ColumnList;
+import org.apache.metron.hbase.client.HBaseClient;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.hbase.ColumnBuilder;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+
+/**
+ * Writes ProfileMeasurement values that can be read during automated testing.
+ */
+public class HBaseProfileWriter implements ProfileWriter {
+
+ private RowKeyBuilder rowKeyBuilder;
+ private ColumnBuilder columnBuilder;
+ private HBaseClient hbaseClient;
+
+ /**
+ * @param rowKeyBuilder Builds the row key for a {@link ProfileMeasurement}.
+ * @param columnBuilder Builds the columns associated with a {@link
ProfileMeasurement}.
+ * @param hbaseClient Writes the {@link ProfileMeasurement} values to HBase.
+ */
+ public HBaseProfileWriter(RowKeyBuilder rowKeyBuilder, ColumnBuilder
columnBuilder, HBaseClient hbaseClient) {
Review comment:
[Per this
comment](https://github.com/apache/metron/pull/1458#discussion_r311217301), I
was able to remove this change from the PR. This should no longer be a concern.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311262976
##
File path:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/ProfilerClientFactory.java
##
@@ -0,0 +1,36 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+package org.apache.metron.profiler.client;
+
+import java.util.Map;
+
+/**
+ * Responsible for creating a {@link ProfilerClient}.
+ */
+public interface ProfilerClientFactory {
Review comment:
Ok, you sold me. See commit 3a29c6500.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311217688
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/ProfileWriter.java
##
@@ -17,109 +17,18 @@
* limitations under the License.
*
*/
-
package org.apache.metron.profiler.client;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hbase.HBaseConfiguration;
-import org.apache.hadoop.hbase.client.Durability;
-import org.apache.hadoop.hbase.client.HTableInterface;
-import org.apache.metron.hbase.HTableProvider;
-import org.apache.metron.hbase.ColumnList;
-import org.apache.metron.hbase.client.LegacyHBaseClient;
import org.apache.metron.profiler.ProfileMeasurement;
-import org.apache.metron.profiler.ProfilePeriod;
-import org.apache.metron.profiler.hbase.ColumnBuilder;
-import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
-import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
-
-import java.util.Collections;
-import java.util.List;
-import java.util.Random;
-import java.util.concurrent.TimeUnit;
-import java.util.function.Function;
/**
- * Writes ProfileMeasurement values that can be read during automated testing.
+ * Responsible for writing {@link ProfileMeasurement} values to persistent
storage.
*/
-public class ProfileWriter {
-
- private RowKeyBuilder rowKeyBuilder;
- private ColumnBuilder columnBuilder;
- private LegacyHBaseClient hbaseClient;
- private HBaseProfilerClient client;
-
- public ProfileWriter(RowKeyBuilder rowKeyBuilder, ColumnBuilder
columnBuilder, HTableInterface table, long periodDurationMillis) {
-this.rowKeyBuilder = rowKeyBuilder;
-this.columnBuilder = columnBuilder;
-this.hbaseClient = new LegacyHBaseClient((c, t) -> table,
table.getConfiguration(), table.getName().getNameAsString());
-this.client = new HBaseProfilerClient(table, rowKeyBuilder, columnBuilder,
periodDurationMillis);
- }
+public interface ProfileWriter {
Review comment:
This is actually legacy code that allowed us to backfill profiles from the
command-line for testing purposes. When the Batch Profiler was introduced this
should have been killed. We don't need the `main` any longer.
The `ProfileGenerator`, `ProfileWriter` and `HBaseProfileWriter` were
refactorings on this legacy functionality when I thought I might need "write"
functionality for the `HBaseClient` API. After additional work I found this
not be necessary.
I removed these unnecessary classes in commit 3e825c1cf.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311217301
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/ProfileGenerator.java
##
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+package org.apache.metron.profiler.client;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.metron.hbase.client.HBaseTableClient;
+import org.apache.metron.hbase.client.HBaseClient;
+import org.apache.metron.hbase.client.HBaseConnectionFactory;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+import org.apache.metron.profiler.hbase.ColumnBuilder;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
+import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+
+/**
+ * Generates a profile by writing multiple sequential {@link
org.apache.metron.profiler.ProfileMeasurement}
+ * values. This is useful for generating profiles to be read during testing.
+ */
+public class ProfileGenerator {
+
+ private ProfileWriter profileWriter;
+
+ public ProfileGenerator(ProfileWriter profileWriter) {
+this.profileWriter = profileWriter;
+ }
+
+ /**
+ * Generates a profile by writing multiple sequential {@link
org.apache.metron.profiler.ProfileMeasurement}
+ * values based on a single prototype.
+ *
+ * @param prototype A prototype for the types of ProfileMeasurements
that should be written.
+ * @param count The number of profile measurements to write.
+ * @param group The name of the group.
+ * @param valueGenerator A function that consumes the previous
ProfileMeasurement value and produces the next.
+ */
+ public void generate(ProfileMeasurement prototype,
+ int count,
+ List group,
+ Function valueGenerator) {
+ProfileMeasurement m = prototype;
+ProfilePeriod period = m.getPeriod();
+for(int i=0; i
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311217301
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/ProfileGenerator.java
##
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+package org.apache.metron.profiler.client;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.metron.hbase.client.HBaseTableClient;
+import org.apache.metron.hbase.client.HBaseClient;
+import org.apache.metron.hbase.client.HBaseConnectionFactory;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+import org.apache.metron.profiler.hbase.ColumnBuilder;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
+import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+
+/**
+ * Generates a profile by writing multiple sequential {@link
org.apache.metron.profiler.ProfileMeasurement}
+ * values. This is useful for generating profiles to be read during testing.
+ */
+public class ProfileGenerator {
+
+ private ProfileWriter profileWriter;
+
+ public ProfileGenerator(ProfileWriter profileWriter) {
+this.profileWriter = profileWriter;
+ }
+
+ /**
+ * Generates a profile by writing multiple sequential {@link
org.apache.metron.profiler.ProfileMeasurement}
+ * values based on a single prototype.
+ *
+ * @param prototype A prototype for the types of ProfileMeasurements
that should be written.
+ * @param count The number of profile measurements to write.
+ * @param group The name of the group.
+ * @param valueGenerator A function that consumes the previous
ProfileMeasurement value and produces the next.
+ */
+ public void generate(ProfileMeasurement prototype,
+ int count,
+ List group,
+ Function valueGenerator) {
+ProfileMeasurement m = prototype;
+ProfilePeriod period = m.getPeriod();
+for(int i=0; i
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311217301
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/ProfileGenerator.java
##
@@ -0,0 +1,106 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+package org.apache.metron.profiler.client;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.metron.hbase.client.HBaseTableClient;
+import org.apache.metron.hbase.client.HBaseClient;
+import org.apache.metron.hbase.client.HBaseConnectionFactory;
+import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
+import org.apache.metron.profiler.hbase.ColumnBuilder;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
+import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+
+/**
+ * Generates a profile by writing multiple sequential {@link
org.apache.metron.profiler.ProfileMeasurement}
+ * values. This is useful for generating profiles to be read during testing.
+ */
+public class ProfileGenerator {
+
+ private ProfileWriter profileWriter;
+
+ public ProfileGenerator(ProfileWriter profileWriter) {
+this.profileWriter = profileWriter;
+ }
+
+ /**
+ * Generates a profile by writing multiple sequential {@link
org.apache.metron.profiler.ProfileMeasurement}
+ * values based on a single prototype.
+ *
+ * @param prototype A prototype for the types of ProfileMeasurements
that should be written.
+ * @param count The number of profile measurements to write.
+ * @param group The name of the group.
+ * @param valueGenerator A function that consumes the previous
ProfileMeasurement value and produces the next.
+ */
+ public void generate(ProfileMeasurement prototype,
+ int count,
+ List group,
+ Function valueGenerator) {
+ProfileMeasurement m = prototype;
+ProfilePeriod period = m.getPeriod();
+for(int i=0; i
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311212993
##
File path:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/ProfilerClientFactory.java
##
@@ -0,0 +1,36 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+package org.apache.metron.profiler.client;
+
+import java.util.Map;
+
+/**
+ * Responsible for creating a {@link ProfilerClient}.
+ */
+public interface ProfilerClientFactory {
Review comment:
I can see the need for an `Extractors`-like class, when
(1) There are many implementations for a user to choose from
(2) A user needs to define via configuration which one to use
(3) Or we want to limit the possible implementations that can be used
In this case, the user never needs to specify which implementation to use
via configuration. There is also only a single implementation that would be
used in production. I don't see where an `Extractors`-like class would help in
this case.
Do you still thing we need this? Am I missing where this might help?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311204989
##
File path:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/VerboseProfile.java
##
@@ -167,56 +167,8 @@ public Object apply(List args, Context context)
throws ParseException {
return view;
}
- /**
- * Creates the ColumnBuilder to use in accessing the profile data.
- * @param global The global configuration.
- */
- private ColumnBuilder getColumnBuilder(Map global) {
-String columnFamily = PROFILER_COLUMN_FAMILY.get(global, String.class);
-return new ValueOnlyColumnBuilder(columnFamily);
- }
-
- /**
- * Creates the ColumnBuilder to use in accessing the profile data.
- * @param global The global configuration.
- */
- private RowKeyBuilder getRowKeyBuilder(Map global) {
-Integer saltDivisor = PROFILER_SALT_DIVISOR.get(global, Integer.class);
-return new SaltyRowKeyBuilder(saltDivisor,
getPeriodDurationInMillis(global), TimeUnit.MILLISECONDS);
- }
-
- /**
- * Create an HBase table used when accessing HBase.
- * @param global The global configuration.
- * @return
- */
- private HTableInterface getTable(Map global) {
-String tableName = PROFILER_HBASE_TABLE.get(global, String.class);
-TableProvider provider = getTableProvider(global);
-try {
- return provider.getTable(HBaseConfiguration.create(), tableName);
-
-} catch (IOException e) {
- throw new IllegalArgumentException(String.format("Unable to access
table: %s", tableName), e);
-}
- }
-
- /**
- * Create the TableProvider to use when accessing HBase.
- * @param global The global configuration.
- */
- private TableProvider getTableProvider(Map global) {
-String clazzName = PROFILER_HBASE_TABLE_PROVIDER.get(global, String.class);
-TableProvider provider;
-try {
- @SuppressWarnings("unchecked")
- Class clazz = (Class)
Class.forName(clazzName);
- provider = clazz.getConstructor().newInstance();
-
-} catch (Exception e) {
- provider = new HTableProvider();
-}
-
-return provider;
+ public VerboseProfile withProfilerClientFactory(ProfilerClientFactory
factory) {
Review comment:
I altered `VerboseProfile` to use the Builder Pattern. See 2464cac00.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311203019
##
File path:
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/SaltyRowKeyBuilder.java
##
@@ -241,10 +241,18 @@ public SaltyRowKeyBuilder(int saltDivisor, long
duration, TimeUnit units) {
}
}
+ public long getPeriodDurationMillis() {
+return periodDurationMillis;
+ }
+
public void withPeriodDuration(long duration, TimeUnit units) {
periodDurationMillis = units.toMillis(duration);
}
+ public int getSaltDivisor() {
Review comment:
I don't have enough context around this to create a JIRA with enough detail
to be useful. I can't even find where salting occurs in Enrichment so as to
document a concrete example. Would you mind creating the JIRA since you have a
better idea of what we might need?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311202048
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
##
@@ -20,437 +20,182 @@
package org.apache.metron.profiler.client.stellar;
-import org.apache.hadoop.hbase.client.HTableInterface;
-import org.apache.metron.hbase.mock.MockHBaseTableProvider;
import org.apache.metron.profiler.ProfileMeasurement;
-import org.apache.metron.profiler.client.ProfileWriter;
-import org.apache.metron.profiler.hbase.ColumnBuilder;
-import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
-import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+import org.apache.metron.profiler.client.ProfilerClient;
import org.apache.metron.stellar.common.DefaultStellarStatefulExecutor;
import org.apache.metron.stellar.common.StellarStatefulExecutor;
import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.functions.resolver.FunctionResolver;
import org.apache.metron.stellar.dsl.functions.resolver.SimpleFunctionResolver;
-import
org.apache.metron.stellar.dsl.functions.resolver.SingletonFunctionResolver;
-import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.TimeUnit;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_COLUMN_FAMILY;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE_PROVIDER;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
/**
- * Tests the GetProfile class.
+ * Tests the 'PROFILE_GET' function in the {@link GetProfile} class.
*/
public class GetProfileTest {
- private static final long periodDuration = 15;
- private static final TimeUnit periodUnits = TimeUnit.MINUTES;
- private static final int saltDivisor = 1000;
- private static final String tableName = "profiler";
- private static final String columnFamily = "P";
private StellarStatefulExecutor executor;
- private Map state;
- private ProfileWriter profileWriter;
- // different values of period and salt divisor, used to test
config_overrides feature
- private static final long periodDuration2 = 1;
- private static final TimeUnit periodUnits2 = TimeUnit.HOURS;
- private static final int saltDivisor2 = 2050;
-
- private T run(String expression, Class clazz) {
-return executor.execute(expression, state, clazz);
+ private FunctionResolver functionResolver;
+ private Map globals;
+ private GetProfile function;
+ private ProfilerClient profilerClient;
+ private List results;
+ private ProfileMeasurement expected;
+
+ private List run(String expression) {
+return executor.execute(expression, new HashMap<>(), List.class);
}
- /**
- * This method sets up the configuration context for both writing profile
data
- * (using profileWriter to mock the complex process of what the Profiler
topology
- * actually does), and then reading that profile data (thereby testing the
PROFILE_GET
- * Stellar client implemented in GetProfile).
- *
- * It runs at @Before time, and sets testclass global variables used by the
writers and readers.
- * The various writers and readers are in each test case, not here.
- *
- * @return void
- */
@Before
public void setup() {
-state = new HashMap<>();
-final HTableInterface table = MockHBaseTableProvider.addToCache(tableName,
columnFamily);
-
-// used to write values to be read during testing
-long periodDurationMillis = TimeUnit.MINUTES.toMillis(15);
-RowKeyBuilder rowKeyBuilder = new SaltyRowKeyBuilder();
-ColumnBuilder columnBuilder = new ValueOnlyColumnBuilder(columnFamily);
-profileWriter = new ProfileWriter(rowKeyBuilder, columnBuilder, table,
periodDurationMillis);
-
-// global properties
-Map global = new HashMap() {{
- put(PROFILER_HBASE_TABLE.getKey(), tableName);
- put(PROFILER_COLUMN_FAMILY.getKey(), columnFamily);
-
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311195074
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
##
@@ -20,437 +20,182 @@
package org.apache.metron.profiler.client.stellar;
-import org.apache.hadoop.hbase.client.HTableInterface;
-import org.apache.metron.hbase.mock.MockHBaseTableProvider;
import org.apache.metron.profiler.ProfileMeasurement;
-import org.apache.metron.profiler.client.ProfileWriter;
-import org.apache.metron.profiler.hbase.ColumnBuilder;
-import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
-import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+import org.apache.metron.profiler.client.ProfilerClient;
import org.apache.metron.stellar.common.DefaultStellarStatefulExecutor;
import org.apache.metron.stellar.common.StellarStatefulExecutor;
import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.functions.resolver.FunctionResolver;
import org.apache.metron.stellar.dsl.functions.resolver.SimpleFunctionResolver;
-import
org.apache.metron.stellar.dsl.functions.resolver.SingletonFunctionResolver;
-import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.TimeUnit;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_COLUMN_FAMILY;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE_PROVIDER;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
/**
- * Tests the GetProfile class.
+ * Tests the 'PROFILE_GET' function in the {@link GetProfile} class.
*/
public class GetProfileTest {
- private static final long periodDuration = 15;
- private static final TimeUnit periodUnits = TimeUnit.MINUTES;
- private static final int saltDivisor = 1000;
- private static final String tableName = "profiler";
- private static final String columnFamily = "P";
private StellarStatefulExecutor executor;
- private Map state;
- private ProfileWriter profileWriter;
- // different values of period and salt divisor, used to test
config_overrides feature
- private static final long periodDuration2 = 1;
- private static final TimeUnit periodUnits2 = TimeUnit.HOURS;
- private static final int saltDivisor2 = 2050;
-
- private T run(String expression, Class clazz) {
-return executor.execute(expression, state, clazz);
+ private FunctionResolver functionResolver;
+ private Map globals;
+ private GetProfile function;
+ private ProfilerClient profilerClient;
+ private List results;
+ private ProfileMeasurement expected;
+
+ private List run(String expression) {
+return executor.execute(expression, new HashMap<>(), List.class);
}
- /**
- * This method sets up the configuration context for both writing profile
data
- * (using profileWriter to mock the complex process of what the Profiler
topology
- * actually does), and then reading that profile data (thereby testing the
PROFILE_GET
- * Stellar client implemented in GetProfile).
- *
- * It runs at @Before time, and sets testclass global variables used by the
writers and readers.
- * The various writers and readers are in each test case, not here.
- *
- * @return void
- */
@Before
public void setup() {
-state = new HashMap<>();
-final HTableInterface table = MockHBaseTableProvider.addToCache(tableName,
columnFamily);
-
-// used to write values to be read during testing
-long periodDurationMillis = TimeUnit.MINUTES.toMillis(15);
-RowKeyBuilder rowKeyBuilder = new SaltyRowKeyBuilder();
-ColumnBuilder columnBuilder = new ValueOnlyColumnBuilder(columnFamily);
-profileWriter = new ProfileWriter(rowKeyBuilder, columnBuilder, table,
periodDurationMillis);
-
-// global properties
-Map global = new HashMap() {{
- put(PROFILER_HBASE_TABLE.getKey(), tableName);
- put(PROFILER_COLUMN_FAMILY.getKey(), columnFamily);
-
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311195074
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
##
@@ -20,437 +20,182 @@
package org.apache.metron.profiler.client.stellar;
-import org.apache.hadoop.hbase.client.HTableInterface;
-import org.apache.metron.hbase.mock.MockHBaseTableProvider;
import org.apache.metron.profiler.ProfileMeasurement;
-import org.apache.metron.profiler.client.ProfileWriter;
-import org.apache.metron.profiler.hbase.ColumnBuilder;
-import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
-import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+import org.apache.metron.profiler.client.ProfilerClient;
import org.apache.metron.stellar.common.DefaultStellarStatefulExecutor;
import org.apache.metron.stellar.common.StellarStatefulExecutor;
import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.functions.resolver.FunctionResolver;
import org.apache.metron.stellar.dsl.functions.resolver.SimpleFunctionResolver;
-import
org.apache.metron.stellar.dsl.functions.resolver.SingletonFunctionResolver;
-import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.TimeUnit;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_COLUMN_FAMILY;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE_PROVIDER;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
/**
- * Tests the GetProfile class.
+ * Tests the 'PROFILE_GET' function in the {@link GetProfile} class.
*/
public class GetProfileTest {
- private static final long periodDuration = 15;
- private static final TimeUnit periodUnits = TimeUnit.MINUTES;
- private static final int saltDivisor = 1000;
- private static final String tableName = "profiler";
- private static final String columnFamily = "P";
private StellarStatefulExecutor executor;
- private Map state;
- private ProfileWriter profileWriter;
- // different values of period and salt divisor, used to test
config_overrides feature
- private static final long periodDuration2 = 1;
- private static final TimeUnit periodUnits2 = TimeUnit.HOURS;
- private static final int saltDivisor2 = 2050;
-
- private T run(String expression, Class clazz) {
-return executor.execute(expression, state, clazz);
+ private FunctionResolver functionResolver;
+ private Map globals;
+ private GetProfile function;
+ private ProfilerClient profilerClient;
+ private List results;
+ private ProfileMeasurement expected;
+
+ private List run(String expression) {
+return executor.execute(expression, new HashMap<>(), List.class);
}
- /**
- * This method sets up the configuration context for both writing profile
data
- * (using profileWriter to mock the complex process of what the Profiler
topology
- * actually does), and then reading that profile data (thereby testing the
PROFILE_GET
- * Stellar client implemented in GetProfile).
- *
- * It runs at @Before time, and sets testclass global variables used by the
writers and readers.
- * The various writers and readers are in each test case, not here.
- *
- * @return void
- */
@Before
public void setup() {
-state = new HashMap<>();
-final HTableInterface table = MockHBaseTableProvider.addToCache(tableName,
columnFamily);
-
-// used to write values to be read during testing
-long periodDurationMillis = TimeUnit.MINUTES.toMillis(15);
-RowKeyBuilder rowKeyBuilder = new SaltyRowKeyBuilder();
-ColumnBuilder columnBuilder = new ValueOnlyColumnBuilder(columnFamily);
-profileWriter = new ProfileWriter(rowKeyBuilder, columnBuilder, table,
periodDurationMillis);
-
-// global properties
-Map global = new HashMap() {{
- put(PROFILER_HBASE_TABLE.getKey(), tableName);
- put(PROFILER_COLUMN_FAMILY.getKey(), columnFamily);
-
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311194672
##
File path:
metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/HBaseWriterFunction.java
##
@@ -137,35 +150,38 @@ public HBaseWriterFunction(Properties properties) {
}
/**
- * Set the {@link TableProvider} using the class name of the provider.
- * @param providerImpl The name of the class.
- * @return
+ * Creates an {@link HBaseConnectionFactory} based on a class name.
+ * @param factoryImpl The class name of an {@link HBaseConnectionFactory}
implementation.
*/
- public HBaseWriterFunction withTableProviderImpl(String providerImpl) {
-this.tableProvider = createTableProvider(providerImpl);
-return this;
- }
-
- /**
- * Creates a TableProvider based on a class name.
- * @param providerImpl The class name of a TableProvider
- */
- private static TableProvider createTableProvider(String providerImpl) {
-LOG.trace("Creating table provider; className={}", providerImpl);
+ private static HBaseConnectionFactory createConnectionFactory(String
factoryImpl) {
+LOG.trace("Creating table provider; className={}", factoryImpl);
// if class name not defined, use a reasonable default
-if(StringUtils.isEmpty(providerImpl) || providerImpl.charAt(0) == '$') {
- return new HTableProvider();
+if(StringUtils.isEmpty(factoryImpl) || factoryImpl.charAt(0) == '$') {
+ return new HBaseConnectionFactory();
}
// instantiate the table provider
-try {
- Class clazz = (Class)
Class.forName(providerImpl);
- return clazz.getConstructor().newInstance();
+return HBaseConnectionFactory.byName(factoryImpl);
+ }
-} catch (InstantiationException | IllegalAccessException |
IllegalStateException |
-InvocationTargetException | NoSuchMethodException |
ClassNotFoundException e) {
- throw new IllegalStateException("Unable to instantiate connector", e);
-}
+ protected HBaseWriterFunction withConnectionFactory(HBaseConnectionFactory
connectionFactory) {
Review comment:
I altered the HBaseWriterFunction to use the Builder Pattern as you
suggested. See fcdb697. Let me know if this is what you're looking for.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311193875
##
File path:
metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/function/HBaseWriterFunctionTest.java
##
@@ -35,61 +41,69 @@
import java.util.Properties;
import java.util.concurrent.TimeUnit;
-import static
org.apache.metron.profiler.spark.BatchProfilerConfig.HBASE_COLUMN_FAMILY;
-import static
org.apache.metron.profiler.spark.BatchProfilerConfig.HBASE_TABLE_NAME;
+import static org.apache.metron.hbase.client.FakeHBaseClient.Mutation;
public class HBaseWriterFunctionTest {
- Properties profilerProperties;
+ private HBaseWriterFunction function;
+ private Properties profilerProperties;
+ private RowKeyBuilder rowKeyBuilder;
+ private ColumnBuilder columnBuilder;
+ private FakeHBaseClient hbaseClient;
+ private HBaseClientFactory hBaseClientFactory;
+
+ private static final JSONObject message = getMessage();
+ private static final String entity = (String) message.get("ip_src_addr");
+ private static final long timestamp = (Long) message.get("timestamp");
+ private static final ProfileConfig profile = getProfile();
@Before
public void setup() {
profilerProperties = getProfilerProperties();
-
-// create a mock table for HBase
-String tableName = HBASE_TABLE_NAME.get(profilerProperties, String.class);
-String columnFamily = HBASE_COLUMN_FAMILY.get(profilerProperties,
String.class);
-MockHBaseTableProvider.addToCache(tableName, columnFamily);
+hbaseClient = new FakeHBaseClient();
+hbaseClient.deleteAll();
+hBaseClientFactory = (x, y, z) -> hbaseClient;
+rowKeyBuilder = new SaltyRowKeyBuilder();
+columnBuilder = new ValueOnlyColumnBuilder();
+function = new HBaseWriterFunction(profilerProperties)
Review comment:
I altered the `HBaseWriterFunction` to use the Builder Pattern as you
suggested. See fcdb697ed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311184981
##
File path:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java
##
@@ -102,94 +90,108 @@
returns="The selected profile measurements."
)
public class GetProfile implements StellarFunction {
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
/**
- * Cached client that can retrieve profile values.
+ * Allows the function to retrieve persisted {@link ProfileMeasurement}
values.
*/
- private ProfilerClient client;
+ private ProfilerClient profilerClient;
/**
- * Cached value of config map actually used to construct the previously
cached client.
+ * Creates the {@link ProfilerClient} used by this function.
*/
- private Map cachedConfigMap = new HashMap(6);
-
- private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private ProfilerClientFactory profilerClientFactory;
/**
- * Initialization. No longer need to do anything in initialization,
- * as all setup is done lazily and cached.
+ * Last known global configuration used to create the {@link
ProfilerClient}. If the
+ * global configuration changes, a new {@link ProfilerClient} needs to be
constructed.
*/
+ private Map lastKnownGlobals = new HashMap<>();
+
+ public GetProfile() {
+profilerClientFactory = new HBaseProfilerClientFactory();
+ }
+
@Override
public void initialize(Context context) {
+Map globals = getGlobals(context);
+profilerClient = profilerClientFactory.create(globals);
}
- /**
- * Is the function initialized?
- */
@Override
public boolean isInitialized() {
-return true;
+return profilerClient != null;
}
- /**
- * Apply the function.
- * @param args The function arguments.
- * @param context
- */
@Override
- public Object apply(List args, Context context) throws
ParseException {
+ public void close() throws IOException {
+if(profilerClient != null) {
+ profilerClient.close();
+}
+ }
+ @Override
+ public Object apply(List args, Context context) throws
ParseException {
+// required arguments
String profile = getArg(0, String.class, args);
String entity = getArg(1, String.class, args);
-Optional> periods = Optional.ofNullable(getArg(2,
List.class, args));
-//Optional arguments
-@SuppressWarnings("unchecked")
-List groups = null;
-Map configOverridesMap = null;
-if (args.size() < 4) {
- // no optional args, so default 'groups' and configOverridesMap remains
null.
- groups = new ArrayList<>(0);
-}
-else if (args.get(3) instanceof List) {
- // correct extensible usage
- groups = getArg(3, List.class, args);
- if (args.size() >= 5) {
-configOverridesMap = getArg(4, Map.class, args);
-if (configOverridesMap.isEmpty()) configOverridesMap = null;
- }
-}
-else {
- // Deprecated "varargs" style usage for groups_list
- // configOverridesMap cannot be specified so it remains null.
- groups = getGroupsArg(3, args);
-}
+List periods = getArg(2, List.class, args);
-Map effectiveConfig = getEffectiveConfig(context,
configOverridesMap);
-Object defaultValue = null;
-//lazily create new profiler client if needed
-if (client == null || !cachedConfigMap.equals(effectiveConfig)) {
- RowKeyBuilder rowKeyBuilder = getRowKeyBuilder(effectiveConfig);
- ColumnBuilder columnBuilder = getColumnBuilder(effectiveConfig);
- HTableInterface table = getTable(effectiveConfig);
- long periodDuration = getPeriodDurationInMillis(effectiveConfig);
- client = new HBaseProfilerClient(table, rowKeyBuilder, columnBuilder,
periodDuration);
- cachedConfigMap = effectiveConfig;
-}
-if(cachedConfigMap != null) {
- defaultValue =
ProfilerClientConfig.PROFILER_DEFAULT_VALUE.get(cachedConfigMap);
+// optional arguments
+List groups = getGroups(args);
+Map overrides = getOverrides(args);
+
+// lazily create new profiler client if needed
+Map effectiveConfig = getEffectiveConfig(context,
overrides);
+if (profilerClient == null || !lastKnownGlobals.equals(effectiveConfig)) {
Review comment:
Sure. https://issues.apache.org/jira/browse/METRON-2204
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311181487
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
##
@@ -20,437 +20,182 @@
package org.apache.metron.profiler.client.stellar;
-import org.apache.hadoop.hbase.client.HTableInterface;
-import org.apache.metron.hbase.mock.MockHBaseTableProvider;
import org.apache.metron.profiler.ProfileMeasurement;
-import org.apache.metron.profiler.client.ProfileWriter;
-import org.apache.metron.profiler.hbase.ColumnBuilder;
-import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
-import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+import org.apache.metron.profiler.client.ProfilerClient;
import org.apache.metron.stellar.common.DefaultStellarStatefulExecutor;
import org.apache.metron.stellar.common.StellarStatefulExecutor;
import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.functions.resolver.FunctionResolver;
import org.apache.metron.stellar.dsl.functions.resolver.SimpleFunctionResolver;
-import
org.apache.metron.stellar.dsl.functions.resolver.SingletonFunctionResolver;
-import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.TimeUnit;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_COLUMN_FAMILY;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE_PROVIDER;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
/**
- * Tests the GetProfile class.
+ * Tests the 'PROFILE_GET' function in the {@link GetProfile} class.
*/
public class GetProfileTest {
- private static final long periodDuration = 15;
- private static final TimeUnit periodUnits = TimeUnit.MINUTES;
- private static final int saltDivisor = 1000;
- private static final String tableName = "profiler";
- private static final String columnFamily = "P";
private StellarStatefulExecutor executor;
- private Map state;
- private ProfileWriter profileWriter;
- // different values of period and salt divisor, used to test
config_overrides feature
- private static final long periodDuration2 = 1;
- private static final TimeUnit periodUnits2 = TimeUnit.HOURS;
- private static final int saltDivisor2 = 2050;
-
- private T run(String expression, Class clazz) {
-return executor.execute(expression, state, clazz);
+ private FunctionResolver functionResolver;
+ private Map globals;
+ private GetProfile function;
+ private ProfilerClient profilerClient;
+ private List results;
+ private ProfileMeasurement expected;
+
+ private List run(String expression) {
+return executor.execute(expression, new HashMap<>(), List.class);
}
- /**
- * This method sets up the configuration context for both writing profile
data
- * (using profileWriter to mock the complex process of what the Profiler
topology
- * actually does), and then reading that profile data (thereby testing the
PROFILE_GET
- * Stellar client implemented in GetProfile).
- *
- * It runs at @Before time, and sets testclass global variables used by the
writers and readers.
- * The various writers and readers are in each test case, not here.
- *
- * @return void
- */
@Before
public void setup() {
-state = new HashMap<>();
-final HTableInterface table = MockHBaseTableProvider.addToCache(tableName,
columnFamily);
-
-// used to write values to be read during testing
-long periodDurationMillis = TimeUnit.MINUTES.toMillis(15);
-RowKeyBuilder rowKeyBuilder = new SaltyRowKeyBuilder();
-ColumnBuilder columnBuilder = new ValueOnlyColumnBuilder(columnFamily);
-profileWriter = new ProfileWriter(rowKeyBuilder, columnBuilder, table,
periodDurationMillis);
-
-// global properties
-Map global = new HashMap() {{
- put(PROFILER_HBASE_TABLE.getKey(), tableName);
- put(PROFILER_COLUMN_FAMILY.getKey(), columnFamily);
-
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311176523
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
##
@@ -20,437 +20,182 @@
package org.apache.metron.profiler.client.stellar;
-import org.apache.hadoop.hbase.client.HTableInterface;
-import org.apache.metron.hbase.mock.MockHBaseTableProvider;
import org.apache.metron.profiler.ProfileMeasurement;
-import org.apache.metron.profiler.client.ProfileWriter;
-import org.apache.metron.profiler.hbase.ColumnBuilder;
-import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
-import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+import org.apache.metron.profiler.client.ProfilerClient;
import org.apache.metron.stellar.common.DefaultStellarStatefulExecutor;
import org.apache.metron.stellar.common.StellarStatefulExecutor;
import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.functions.resolver.FunctionResolver;
import org.apache.metron.stellar.dsl.functions.resolver.SimpleFunctionResolver;
-import
org.apache.metron.stellar.dsl.functions.resolver.SingletonFunctionResolver;
-import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.TimeUnit;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_COLUMN_FAMILY;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE_PROVIDER;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
/**
- * Tests the GetProfile class.
+ * Tests the 'PROFILE_GET' function in the {@link GetProfile} class.
*/
public class GetProfileTest {
- private static final long periodDuration = 15;
- private static final TimeUnit periodUnits = TimeUnit.MINUTES;
- private static final int saltDivisor = 1000;
- private static final String tableName = "profiler";
- private static final String columnFamily = "P";
private StellarStatefulExecutor executor;
- private Map state;
- private ProfileWriter profileWriter;
- // different values of period and salt divisor, used to test
config_overrides feature
- private static final long periodDuration2 = 1;
- private static final TimeUnit periodUnits2 = TimeUnit.HOURS;
- private static final int saltDivisor2 = 2050;
-
- private T run(String expression, Class clazz) {
-return executor.execute(expression, state, clazz);
+ private FunctionResolver functionResolver;
+ private Map globals;
+ private GetProfile function;
+ private ProfilerClient profilerClient;
+ private List results;
+ private ProfileMeasurement expected;
+
+ private List run(String expression) {
+return executor.execute(expression, new HashMap<>(), List.class);
}
- /**
- * This method sets up the configuration context for both writing profile
data
- * (using profileWriter to mock the complex process of what the Profiler
topology
- * actually does), and then reading that profile data (thereby testing the
PROFILE_GET
- * Stellar client implemented in GetProfile).
- *
- * It runs at @Before time, and sets testclass global variables used by the
writers and readers.
- * The various writers and readers are in each test case, not here.
- *
- * @return void
- */
@Before
public void setup() {
-state = new HashMap<>();
-final HTableInterface table = MockHBaseTableProvider.addToCache(tableName,
columnFamily);
-
-// used to write values to be read during testing
-long periodDurationMillis = TimeUnit.MINUTES.toMillis(15);
-RowKeyBuilder rowKeyBuilder = new SaltyRowKeyBuilder();
-ColumnBuilder columnBuilder = new ValueOnlyColumnBuilder(columnFamily);
-profileWriter = new ProfileWriter(rowKeyBuilder, columnBuilder, table,
periodDurationMillis);
-
-// global properties
-Map global = new HashMap() {{
- put(PROFILER_HBASE_TABLE.getKey(), tableName);
- put(PROFILER_COLUMN_FAMILY.getKey(), columnFamily);
-
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311176523
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
##
@@ -20,437 +20,182 @@
package org.apache.metron.profiler.client.stellar;
-import org.apache.hadoop.hbase.client.HTableInterface;
-import org.apache.metron.hbase.mock.MockHBaseTableProvider;
import org.apache.metron.profiler.ProfileMeasurement;
-import org.apache.metron.profiler.client.ProfileWriter;
-import org.apache.metron.profiler.hbase.ColumnBuilder;
-import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
-import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+import org.apache.metron.profiler.client.ProfilerClient;
import org.apache.metron.stellar.common.DefaultStellarStatefulExecutor;
import org.apache.metron.stellar.common.StellarStatefulExecutor;
import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.functions.resolver.FunctionResolver;
import org.apache.metron.stellar.dsl.functions.resolver.SimpleFunctionResolver;
-import
org.apache.metron.stellar.dsl.functions.resolver.SingletonFunctionResolver;
-import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.TimeUnit;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_COLUMN_FAMILY;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE_PROVIDER;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
/**
- * Tests the GetProfile class.
+ * Tests the 'PROFILE_GET' function in the {@link GetProfile} class.
*/
public class GetProfileTest {
- private static final long periodDuration = 15;
- private static final TimeUnit periodUnits = TimeUnit.MINUTES;
- private static final int saltDivisor = 1000;
- private static final String tableName = "profiler";
- private static final String columnFamily = "P";
private StellarStatefulExecutor executor;
- private Map state;
- private ProfileWriter profileWriter;
- // different values of period and salt divisor, used to test
config_overrides feature
- private static final long periodDuration2 = 1;
- private static final TimeUnit periodUnits2 = TimeUnit.HOURS;
- private static final int saltDivisor2 = 2050;
-
- private T run(String expression, Class clazz) {
-return executor.execute(expression, state, clazz);
+ private FunctionResolver functionResolver;
+ private Map globals;
+ private GetProfile function;
+ private ProfilerClient profilerClient;
+ private List results;
+ private ProfileMeasurement expected;
+
+ private List run(String expression) {
+return executor.execute(expression, new HashMap<>(), List.class);
}
- /**
- * This method sets up the configuration context for both writing profile
data
- * (using profileWriter to mock the complex process of what the Profiler
topology
- * actually does), and then reading that profile data (thereby testing the
PROFILE_GET
- * Stellar client implemented in GetProfile).
- *
- * It runs at @Before time, and sets testclass global variables used by the
writers and readers.
- * The various writers and readers are in each test case, not here.
- *
- * @return void
- */
@Before
public void setup() {
-state = new HashMap<>();
-final HTableInterface table = MockHBaseTableProvider.addToCache(tableName,
columnFamily);
-
-// used to write values to be read during testing
-long periodDurationMillis = TimeUnit.MINUTES.toMillis(15);
-RowKeyBuilder rowKeyBuilder = new SaltyRowKeyBuilder();
-ColumnBuilder columnBuilder = new ValueOnlyColumnBuilder(columnFamily);
-profileWriter = new ProfileWriter(rowKeyBuilder, columnBuilder, table,
periodDurationMillis);
-
-// global properties
-Map global = new HashMap() {{
- put(PROFILER_HBASE_TABLE.getKey(), tableName);
- put(PROFILER_COLUMN_FAMILY.getKey(), columnFamily);
-
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2 URL: https://github.com/apache/metron/pull/1458#discussion_r311169680 ## File path: metron-analytics/metron-profiler-storm/src/main/config/profiler.properties ## @@ -61,6 +61,8 @@ profiler.hbase.table=profiler profiler.hbase.column.family=P profiler.hbase.batch=10 profiler.hbase.flush.interval.seconds=30 +profiler.hbase.client.factory=org.apache.metron.hbase.client.HBaseTableClientFactory Review comment: The `HBaseTableClient` is an `HBaseClient` that uses HBase's `Table` abstraction in its implementation. And that is as opposed to being implemented in another way, like for example using HBase's `BufferedMutator` abstraction. This is how I landed on the name `HBaseTableClient`. Actually in earlier versions of this upgrade there was some legacy code that was using a `BufferedMutator`. So I had to implement an `HBaseClient` using a `BufferedMutator` to have feature parity. I called it something like `BufferedMutatorHBaseClient`. After putting in that work, I realized that the legacy code using the `BufferedMutator` was completely dead code, so I didn't need this implementation after all. If we can find a better name for this class, I'd be happy to change in a follow-on PR. I understand the source of confusion since "table" is an important logical concept in HBase. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2 URL: https://github.com/apache/metron/pull/1458#discussion_r311169680 ## File path: metron-analytics/metron-profiler-storm/src/main/config/profiler.properties ## @@ -61,6 +61,8 @@ profiler.hbase.table=profiler profiler.hbase.column.family=P profiler.hbase.batch=10 profiler.hbase.flush.interval.seconds=30 +profiler.hbase.client.factory=org.apache.metron.hbase.client.HBaseTableClientFactory Review comment: The `HBaseTableClient` is an `HBaseClient` that uses HBase's `Table` abstraction in its implementation. And, for example, that is as opposed to being implemented in another way, like using HBase's `BufferedMutator` abstraction. This is how I landed on the name `HBaseTableClient`. Actually in earlier versions of this upgrade there was some legacy code that was using a `BufferedMutator`. So I had to implement an `HBaseClient` using a `BufferedMutator` to have feature parity. I called it something like `BufferedMutatorHBaseClient`. After putting in that work, I realized that the legacy code using the `BufferedMutator` was completely dead code, so I didn't need this implementation after all. If we can find a better name for this class, I'd be happy to change in a follow-on PR. I understand the source of confusion since "table" is an important logical concept in HBase. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2 URL: https://github.com/apache/metron/pull/1458#discussion_r311169680 ## File path: metron-analytics/metron-profiler-storm/src/main/config/profiler.properties ## @@ -61,6 +61,8 @@ profiler.hbase.table=profiler profiler.hbase.column.family=P profiler.hbase.batch=10 profiler.hbase.flush.interval.seconds=30 +profiler.hbase.client.factory=org.apache.metron.hbase.client.HBaseTableClientFactory Review comment: The `HBaseTableClient` is an `HBaseClient` that uses HBase's `Table` abstraction in its implementation. And, for example, that is as opposed to being implemented in another way, like using HBase's `BufferedMutator` abstraction. This is how I landed on the name `HBaseTableClient`. Actually in earlier versions of this upgrade there was some legacy code that was using a `BufferedMutator`. So I had to implement an `HBaseClient` using a `BufferedMutator` to have feature parity. I called it something like `BufferedMutatorHBaseClient`. After putting in that work, I realized that the legacy code using the `BufferedMutator` was completely dead code, so I didn't need this implementation after all. If we can find a better name for this class, I'd be happy to change in a follow-on PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311165058
##
File path:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/VerboseProfile.java
##
@@ -122,35 +129,28 @@ public Object apply(List args, Context context)
throws ParseException {
groups = getArg(3, List.class, args);
}
-// get globals from the context
-Map globals = (Map)
context.getCapability(GLOBAL_CONFIG)
-.orElse(Collections.emptyMap());
-
-// lazily create the profiler client, if needed
Review comment:
(1) I added the comment in the latest commit.
(2) See [METRON-2204](https://issues.apache.org/jira/browse/METRON-2204)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311156658
##
File path:
metron-analytics/metron-profiler-storm/src/test/java/org/apache/metron/profiler/storm/integration/ProfilerIntegrationTest.java
##
@@ -503,21 +496,29 @@ public void setup() {
put(PROFILER_SALT_DIVISOR.getKey(), saltDivisor);
}};
+Context context = new Context.Builder()
+.with(Context.Capabilities.GLOBAL_CONFIG, () -> global)
+.build();
+
+// create the GET_PROFILE function
+GetProfile getProfileFunction = new GetProfile()
+.withProfilerClientFactory(new HBaseProfilerClientFactory(new
FakeHBaseClientFactory()));
Review comment:
Correct. You will see that change in #1470 .
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311154623
##
File path:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java
##
@@ -102,94 +90,108 @@
returns="The selected profile measurements."
)
public class GetProfile implements StellarFunction {
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
/**
- * Cached client that can retrieve profile values.
+ * Allows the function to retrieve persisted {@link ProfileMeasurement}
values.
*/
- private ProfilerClient client;
+ private ProfilerClient profilerClient;
/**
- * Cached value of config map actually used to construct the previously
cached client.
+ * Creates the {@link ProfilerClient} used by this function.
*/
- private Map cachedConfigMap = new HashMap(6);
-
- private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private ProfilerClientFactory profilerClientFactory;
/**
- * Initialization. No longer need to do anything in initialization,
- * as all setup is done lazily and cached.
+ * Last known global configuration used to create the {@link
ProfilerClient}. If the
+ * global configuration changes, a new {@link ProfilerClient} needs to be
constructed.
*/
+ private Map lastKnownGlobals = new HashMap<>();
+
+ public GetProfile() {
+profilerClientFactory = new HBaseProfilerClientFactory();
+ }
+
@Override
public void initialize(Context context) {
+Map globals = getGlobals(context);
+profilerClient = profilerClientFactory.create(globals);
}
- /**
- * Is the function initialized?
- */
@Override
public boolean isInitialized() {
-return true;
+return profilerClient != null;
}
- /**
- * Apply the function.
- * @param args The function arguments.
- * @param context
- */
@Override
- public Object apply(List args, Context context) throws
ParseException {
+ public void close() throws IOException {
+if(profilerClient != null) {
+ profilerClient.close();
+}
+ }
+ @Override
+ public Object apply(List args, Context context) throws
ParseException {
+// required arguments
String profile = getArg(0, String.class, args);
String entity = getArg(1, String.class, args);
-Optional> periods = Optional.ofNullable(getArg(2,
List.class, args));
-//Optional arguments
-@SuppressWarnings("unchecked")
-List groups = null;
-Map configOverridesMap = null;
-if (args.size() < 4) {
- // no optional args, so default 'groups' and configOverridesMap remains
null.
- groups = new ArrayList<>(0);
-}
-else if (args.get(3) instanceof List) {
- // correct extensible usage
- groups = getArg(3, List.class, args);
- if (args.size() >= 5) {
-configOverridesMap = getArg(4, Map.class, args);
-if (configOverridesMap.isEmpty()) configOverridesMap = null;
- }
-}
-else {
- // Deprecated "varargs" style usage for groups_list
- // configOverridesMap cannot be specified so it remains null.
- groups = getGroupsArg(3, args);
-}
+List periods = getArg(2, List.class, args);
-Map effectiveConfig = getEffectiveConfig(context,
configOverridesMap);
-Object defaultValue = null;
-//lazily create new profiler client if needed
-if (client == null || !cachedConfigMap.equals(effectiveConfig)) {
- RowKeyBuilder rowKeyBuilder = getRowKeyBuilder(effectiveConfig);
- ColumnBuilder columnBuilder = getColumnBuilder(effectiveConfig);
- HTableInterface table = getTable(effectiveConfig);
- long periodDuration = getPeriodDurationInMillis(effectiveConfig);
- client = new HBaseProfilerClient(table, rowKeyBuilder, columnBuilder,
periodDuration);
- cachedConfigMap = effectiveConfig;
-}
-if(cachedConfigMap != null) {
- defaultValue =
ProfilerClientConfig.PROFILER_DEFAULT_VALUE.get(cachedConfigMap);
+// optional arguments
+List groups = getGroups(args);
+Map overrides = getOverrides(args);
+
+// lazily create new profiler client if needed
+Map effectiveConfig = getEffectiveConfig(context,
overrides);
+if (profilerClient == null || !lastKnownGlobals.equals(effectiveConfig)) {
+ profilerClient = profilerClientFactory.create(effectiveConfig);
+ lastKnownGlobals = effectiveConfig;
}
-List measurements = client.fetch(Object.class,
profile, entity, groups,
-periods.orElse(new ArrayList<>(0)),
Optional.ofNullable(defaultValue));
+// is there a default value?
+Optional defaultValue = Optional.empty();
+if(effectiveConfig != null) {
+ defaultValue =
Optional.ofNullable(PROFILER_DEFAULT_VALUE.get(effectiveConfig));
+}
// return only the value
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311098015
##
File path:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/VerboseProfile.java
##
@@ -122,35 +129,28 @@ public Object apply(List args, Context context)
throws ParseException {
groups = getArg(3, List.class, args);
}
-// get globals from the context
-Map globals = (Map)
context.getCapability(GLOBAL_CONFIG)
-.orElse(Collections.emptyMap());
-
-// lazily create the profiler client, if needed
Review comment:
The `PROFILE_VERBOSE` function has lazy initialization, but will not respond
to changes in the global config like `PROFILE_GET` does. This is how I
initially implemented `PROFILE_VERBOSE` in #1292 (right or wrong).
I believe the thought process was `PROFILE_VERBOSE` is not a "production"
function and is only useful for troubleshooting in the REPL. So if you change
the global config, you're going to have to restart the REPL. But to your point,
why have different behaviors for these very similar functions and violate
[POLA](https://en.wikipedia.org/wiki/Principle_of_least_astonishment)?
How about for this PR, I ...
(1) Add comments describing the disparity.
(2) Create a JIRA to alter `PROFILE_VERBOSE` to behave like `PROFILE_GET`
and respond to changes in the global configuration.
I think it is a worthwhile improvement, I'm just not sure how much code
will need to change to make this happen on this already large PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311083234
##
File path:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java
##
@@ -102,94 +90,108 @@
returns="The selected profile measurements."
)
public class GetProfile implements StellarFunction {
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
/**
- * Cached client that can retrieve profile values.
+ * Allows the function to retrieve persisted {@link ProfileMeasurement}
values.
*/
- private ProfilerClient client;
+ private ProfilerClient profilerClient;
/**
- * Cached value of config map actually used to construct the previously
cached client.
+ * Creates the {@link ProfilerClient} used by this function.
*/
- private Map cachedConfigMap = new HashMap(6);
-
- private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private ProfilerClientFactory profilerClientFactory;
/**
- * Initialization. No longer need to do anything in initialization,
- * as all setup is done lazily and cached.
+ * Last known global configuration used to create the {@link
ProfilerClient}. If the
+ * global configuration changes, a new {@link ProfilerClient} needs to be
constructed.
*/
+ private Map lastKnownGlobals = new HashMap<>();
+
+ public GetProfile() {
+profilerClientFactory = new HBaseProfilerClientFactory();
+ }
+
@Override
public void initialize(Context context) {
+Map globals = getGlobals(context);
+profilerClient = profilerClientFactory.create(globals);
}
- /**
- * Is the function initialized?
- */
@Override
public boolean isInitialized() {
-return true;
+return profilerClient != null;
}
- /**
- * Apply the function.
- * @param args The function arguments.
- * @param context
- */
@Override
- public Object apply(List args, Context context) throws
ParseException {
+ public void close() throws IOException {
+if(profilerClient != null) {
+ profilerClient.close();
+}
+ }
+ @Override
+ public Object apply(List args, Context context) throws
ParseException {
+// required arguments
String profile = getArg(0, String.class, args);
String entity = getArg(1, String.class, args);
-Optional> periods = Optional.ofNullable(getArg(2,
List.class, args));
-//Optional arguments
-@SuppressWarnings("unchecked")
-List groups = null;
-Map configOverridesMap = null;
-if (args.size() < 4) {
- // no optional args, so default 'groups' and configOverridesMap remains
null.
- groups = new ArrayList<>(0);
-}
-else if (args.get(3) instanceof List) {
- // correct extensible usage
- groups = getArg(3, List.class, args);
- if (args.size() >= 5) {
-configOverridesMap = getArg(4, Map.class, args);
-if (configOverridesMap.isEmpty()) configOverridesMap = null;
- }
-}
-else {
- // Deprecated "varargs" style usage for groups_list
- // configOverridesMap cannot be specified so it remains null.
- groups = getGroupsArg(3, args);
-}
+List periods = getArg(2, List.class, args);
-Map effectiveConfig = getEffectiveConfig(context,
configOverridesMap);
-Object defaultValue = null;
-//lazily create new profiler client if needed
-if (client == null || !cachedConfigMap.equals(effectiveConfig)) {
- RowKeyBuilder rowKeyBuilder = getRowKeyBuilder(effectiveConfig);
- ColumnBuilder columnBuilder = getColumnBuilder(effectiveConfig);
- HTableInterface table = getTable(effectiveConfig);
- long periodDuration = getPeriodDurationInMillis(effectiveConfig);
- client = new HBaseProfilerClient(table, rowKeyBuilder, columnBuilder,
periodDuration);
- cachedConfigMap = effectiveConfig;
-}
-if(cachedConfigMap != null) {
- defaultValue =
ProfilerClientConfig.PROFILER_DEFAULT_VALUE.get(cachedConfigMap);
+// optional arguments
+List groups = getGroups(args);
+Map overrides = getOverrides(args);
+
+// lazily create new profiler client if needed
+Map effectiveConfig = getEffectiveConfig(context,
overrides);
+if (profilerClient == null || !lastKnownGlobals.equals(effectiveConfig)) {
Review comment:
Yes, I believe the logic was intended to create a new `ProfilerClient`
should the global configuration change. This is legacy code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311075260
##
File path:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java
##
@@ -102,94 +90,108 @@
returns="The selected profile measurements."
)
public class GetProfile implements StellarFunction {
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
/**
- * Cached client that can retrieve profile values.
+ * Allows the function to retrieve persisted {@link ProfileMeasurement}
values.
*/
- private ProfilerClient client;
+ private ProfilerClient profilerClient;
/**
- * Cached value of config map actually used to construct the previously
cached client.
+ * Creates the {@link ProfilerClient} used by this function.
*/
- private Map cachedConfigMap = new HashMap(6);
-
- private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private ProfilerClientFactory profilerClientFactory;
/**
- * Initialization. No longer need to do anything in initialization,
- * as all setup is done lazily and cached.
+ * Last known global configuration used to create the {@link
ProfilerClient}. If the
+ * global configuration changes, a new {@link ProfilerClient} needs to be
constructed.
*/
+ private Map lastKnownGlobals = new HashMap<>();
+
+ public GetProfile() {
+profilerClientFactory = new HBaseProfilerClientFactory();
+ }
+
@Override
public void initialize(Context context) {
+Map globals = getGlobals(context);
+profilerClient = profilerClientFactory.create(globals);
}
- /**
- * Is the function initialized?
- */
@Override
public boolean isInitialized() {
-return true;
+return profilerClient != null;
}
- /**
- * Apply the function.
- * @param args The function arguments.
- * @param context
- */
@Override
- public Object apply(List args, Context context) throws
ParseException {
+ public void close() throws IOException {
+if(profilerClient != null) {
+ profilerClient.close();
+}
+ }
+ @Override
+ public Object apply(List args, Context context) throws
ParseException {
+// required arguments
String profile = getArg(0, String.class, args);
String entity = getArg(1, String.class, args);
-Optional> periods = Optional.ofNullable(getArg(2,
List.class, args));
-//Optional arguments
-@SuppressWarnings("unchecked")
-List groups = null;
-Map configOverridesMap = null;
-if (args.size() < 4) {
- // no optional args, so default 'groups' and configOverridesMap remains
null.
- groups = new ArrayList<>(0);
-}
-else if (args.get(3) instanceof List) {
- // correct extensible usage
- groups = getArg(3, List.class, args);
- if (args.size() >= 5) {
-configOverridesMap = getArg(4, Map.class, args);
-if (configOverridesMap.isEmpty()) configOverridesMap = null;
- }
-}
-else {
- // Deprecated "varargs" style usage for groups_list
- // configOverridesMap cannot be specified so it remains null.
- groups = getGroupsArg(3, args);
-}
+List periods = getArg(2, List.class, args);
-Map effectiveConfig = getEffectiveConfig(context,
configOverridesMap);
-Object defaultValue = null;
-//lazily create new profiler client if needed
-if (client == null || !cachedConfigMap.equals(effectiveConfig)) {
- RowKeyBuilder rowKeyBuilder = getRowKeyBuilder(effectiveConfig);
- ColumnBuilder columnBuilder = getColumnBuilder(effectiveConfig);
- HTableInterface table = getTable(effectiveConfig);
- long periodDuration = getPeriodDurationInMillis(effectiveConfig);
- client = new HBaseProfilerClient(table, rowKeyBuilder, columnBuilder,
periodDuration);
- cachedConfigMap = effectiveConfig;
-}
-if(cachedConfigMap != null) {
- defaultValue =
ProfilerClientConfig.PROFILER_DEFAULT_VALUE.get(cachedConfigMap);
+// optional arguments
+List groups = getGroups(args);
+Map overrides = getOverrides(args);
+
+// lazily create new profiler client if needed
+Map effectiveConfig = getEffectiveConfig(context,
overrides);
+if (profilerClient == null || !lastKnownGlobals.equals(effectiveConfig)) {
+ profilerClient = profilerClientFactory.create(effectiveConfig);
+ lastKnownGlobals = effectiveConfig;
}
-List measurements = client.fetch(Object.class,
profile, entity, groups,
-periods.orElse(new ArrayList<>(0)),
Optional.ofNullable(defaultValue));
+// is there a default value?
+Optional defaultValue = Optional.empty();
+if(effectiveConfig != null) {
+ defaultValue =
Optional.ofNullable(PROFILER_DEFAULT_VALUE.get(effectiveConfig));
+}
// return only the value
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2 URL: https://github.com/apache/metron/pull/1458#discussion_r311081338 ## File path: dependencies_with_url.csv ## @@ -146,6 +146,7 @@ com.codahale.metrics:metrics-core:jar:3.0.2:compile,MIT,https://github.com/codah com.codahale.metrics:metrics-graphite:jar:3.0.2:compile,MIT,https://github.com/codahale/metrics com.esotericsoftware.reflectasm:reflectasm:jar:shaded:1.07:compile,BSD,https://github.com/EsotericSoftware/reflectasm com.fasterxml.jackson.core:jackson-annotations:jar:2.2.3:compile,ASLv2,http://wiki.fasterxml.com/JacksonHome +com.fasterxml.jackson.core:jackson-annotations:jar:2.7.0:compile,ASLv2,http://wiki.fasterxml.com/JacksonHome Review comment: Maybe this is the cause [of the issue](https://github.com/apache/metron/pull/1458#issuecomment-516178822) @merrimanr found during testing which I was not able to replicate. I will track this down. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311063115
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
##
@@ -20,437 +20,182 @@
package org.apache.metron.profiler.client.stellar;
-import org.apache.hadoop.hbase.client.HTableInterface;
-import org.apache.metron.hbase.mock.MockHBaseTableProvider;
import org.apache.metron.profiler.ProfileMeasurement;
-import org.apache.metron.profiler.client.ProfileWriter;
-import org.apache.metron.profiler.hbase.ColumnBuilder;
-import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
-import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+import org.apache.metron.profiler.client.ProfilerClient;
import org.apache.metron.stellar.common.DefaultStellarStatefulExecutor;
import org.apache.metron.stellar.common.StellarStatefulExecutor;
import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.functions.resolver.FunctionResolver;
import org.apache.metron.stellar.dsl.functions.resolver.SimpleFunctionResolver;
-import
org.apache.metron.stellar.dsl.functions.resolver.SingletonFunctionResolver;
-import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.TimeUnit;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_COLUMN_FAMILY;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE_PROVIDER;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
/**
- * Tests the GetProfile class.
+ * Tests the 'PROFILE_GET' function in the {@link GetProfile} class.
Review comment:
I often find myself trying to find the implementation and test classes of
Stellar functions. For example, "Where is the code that implements/tests
`PROFILE_GET`?"
Due to the way our `StellarFunction` annotation works (namespace="PROFILE",
function="GET"), there is not a simple way to just search for `PROFILE_GET`.
The search won't hit on anything.
That is why I though to add the phrase `PROFILE_GET` here, so the test class
can be more easily found.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2 URL: https://github.com/apache/metron/pull/1458#discussion_r311072877 ## File path: metron-analytics/metron-profiler-storm/src/main/config/profiler.properties ## @@ -61,6 +61,8 @@ profiler.hbase.table=profiler profiler.hbase.column.family=P profiler.hbase.batch=10 profiler.hbase.flush.interval.seconds=30 +profiler.hbase.client.factory=org.apache.metron.hbase.client.HBaseTableClientFactory Review comment: * `HBaseConnectionFactory` creates connections to HBase. * `HBaseClientFactory` creates the `HBaseClient`s. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2 URL: https://github.com/apache/metron/pull/1458#discussion_r311072877 ## File path: metron-analytics/metron-profiler-storm/src/main/config/profiler.properties ## @@ -61,6 +61,8 @@ profiler.hbase.table=profiler profiler.hbase.column.family=P profiler.hbase.batch=10 profiler.hbase.flush.interval.seconds=30 +profiler.hbase.client.factory=org.apache.metron.hbase.client.HBaseTableClientFactory Review comment: * `HBaseConnectionFactory` creates connections to HBase. * `HBaseClientFactory` creates `HBaseClient`s. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311058843
##
File path:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/ProfilerClientFactory.java
##
@@ -0,0 +1,36 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+package org.apache.metron.profiler.client;
+
+import java.util.Map;
+
+/**
+ * Responsible for creating a {@link ProfilerClient}.
+ */
+public interface ProfilerClientFactory {
Review comment:
What are you suggesting? The "factory" in that example is called
`ExtractorCreator`. Do you think this should be called
`ProfilerClientCreator`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311075260
##
File path:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java
##
@@ -102,94 +90,108 @@
returns="The selected profile measurements."
)
public class GetProfile implements StellarFunction {
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
/**
- * Cached client that can retrieve profile values.
+ * Allows the function to retrieve persisted {@link ProfileMeasurement}
values.
*/
- private ProfilerClient client;
+ private ProfilerClient profilerClient;
/**
- * Cached value of config map actually used to construct the previously
cached client.
+ * Creates the {@link ProfilerClient} used by this function.
*/
- private Map cachedConfigMap = new HashMap(6);
-
- private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private ProfilerClientFactory profilerClientFactory;
/**
- * Initialization. No longer need to do anything in initialization,
- * as all setup is done lazily and cached.
+ * Last known global configuration used to create the {@link
ProfilerClient}. If the
+ * global configuration changes, a new {@link ProfilerClient} needs to be
constructed.
*/
+ private Map lastKnownGlobals = new HashMap<>();
+
+ public GetProfile() {
+profilerClientFactory = new HBaseProfilerClientFactory();
+ }
+
@Override
public void initialize(Context context) {
+Map globals = getGlobals(context);
+profilerClient = profilerClientFactory.create(globals);
}
- /**
- * Is the function initialized?
- */
@Override
public boolean isInitialized() {
-return true;
+return profilerClient != null;
}
- /**
- * Apply the function.
- * @param args The function arguments.
- * @param context
- */
@Override
- public Object apply(List args, Context context) throws
ParseException {
+ public void close() throws IOException {
+if(profilerClient != null) {
+ profilerClient.close();
+}
+ }
+ @Override
+ public Object apply(List args, Context context) throws
ParseException {
+// required arguments
String profile = getArg(0, String.class, args);
String entity = getArg(1, String.class, args);
-Optional> periods = Optional.ofNullable(getArg(2,
List.class, args));
-//Optional arguments
-@SuppressWarnings("unchecked")
-List groups = null;
-Map configOverridesMap = null;
-if (args.size() < 4) {
- // no optional args, so default 'groups' and configOverridesMap remains
null.
- groups = new ArrayList<>(0);
-}
-else if (args.get(3) instanceof List) {
- // correct extensible usage
- groups = getArg(3, List.class, args);
- if (args.size() >= 5) {
-configOverridesMap = getArg(4, Map.class, args);
-if (configOverridesMap.isEmpty()) configOverridesMap = null;
- }
-}
-else {
- // Deprecated "varargs" style usage for groups_list
- // configOverridesMap cannot be specified so it remains null.
- groups = getGroupsArg(3, args);
-}
+List periods = getArg(2, List.class, args);
-Map effectiveConfig = getEffectiveConfig(context,
configOverridesMap);
-Object defaultValue = null;
-//lazily create new profiler client if needed
-if (client == null || !cachedConfigMap.equals(effectiveConfig)) {
- RowKeyBuilder rowKeyBuilder = getRowKeyBuilder(effectiveConfig);
- ColumnBuilder columnBuilder = getColumnBuilder(effectiveConfig);
- HTableInterface table = getTable(effectiveConfig);
- long periodDuration = getPeriodDurationInMillis(effectiveConfig);
- client = new HBaseProfilerClient(table, rowKeyBuilder, columnBuilder,
periodDuration);
- cachedConfigMap = effectiveConfig;
-}
-if(cachedConfigMap != null) {
- defaultValue =
ProfilerClientConfig.PROFILER_DEFAULT_VALUE.get(cachedConfigMap);
+// optional arguments
+List groups = getGroups(args);
+Map overrides = getOverrides(args);
+
+// lazily create new profiler client if needed
+Map effectiveConfig = getEffectiveConfig(context,
overrides);
+if (profilerClient == null || !lastKnownGlobals.equals(effectiveConfig)) {
+ profilerClient = profilerClientFactory.create(effectiveConfig);
+ lastKnownGlobals = effectiveConfig;
}
-List measurements = client.fetch(Object.class,
profile, entity, groups,
-periods.orElse(new ArrayList<>(0)),
Optional.ofNullable(defaultValue));
+// is there a default value?
+Optional defaultValue = Optional.empty();
+if(effectiveConfig != null) {
+ defaultValue =
Optional.ofNullable(PROFILER_DEFAULT_VALUE.get(effectiveConfig));
+}
// return only the value
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2 URL: https://github.com/apache/metron/pull/1458#discussion_r311072877 ## File path: metron-analytics/metron-profiler-storm/src/main/config/profiler.properties ## @@ -61,6 +61,8 @@ profiler.hbase.table=profiler profiler.hbase.column.family=P profiler.hbase.batch=10 profiler.hbase.flush.interval.seconds=30 +profiler.hbase.client.factory=org.apache.metron.hbase.client.HBaseTableClientFactory Review comment: * A `HBaseConnectionFactory` creates connections to HBase. * A `HBaseClientFactory` creates `HBaseClient`s. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311064624
##
File path:
metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/hbase/SaltyRowKeyBuilder.java
##
@@ -241,10 +241,18 @@ public SaltyRowKeyBuilder(int saltDivisor, long
duration, TimeUnit units) {
}
}
+ public long getPeriodDurationMillis() {
+return periodDurationMillis;
+ }
+
public void withPeriodDuration(long duration, TimeUnit units) {
periodDurationMillis = units.toMillis(duration);
}
+ public int getSaltDivisor() {
Review comment:
This is what generates the row keys specifically for profiles stored in
HBase. It does not relate to Enrichments at all.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311063115
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
##
@@ -20,437 +20,182 @@
package org.apache.metron.profiler.client.stellar;
-import org.apache.hadoop.hbase.client.HTableInterface;
-import org.apache.metron.hbase.mock.MockHBaseTableProvider;
import org.apache.metron.profiler.ProfileMeasurement;
-import org.apache.metron.profiler.client.ProfileWriter;
-import org.apache.metron.profiler.hbase.ColumnBuilder;
-import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
-import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+import org.apache.metron.profiler.client.ProfilerClient;
import org.apache.metron.stellar.common.DefaultStellarStatefulExecutor;
import org.apache.metron.stellar.common.StellarStatefulExecutor;
import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.functions.resolver.FunctionResolver;
import org.apache.metron.stellar.dsl.functions.resolver.SimpleFunctionResolver;
-import
org.apache.metron.stellar.dsl.functions.resolver.SingletonFunctionResolver;
-import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.TimeUnit;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_COLUMN_FAMILY;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE_PROVIDER;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
/**
- * Tests the GetProfile class.
+ * Tests the 'PROFILE_GET' function in the {@link GetProfile} class.
Review comment:
I often find myself trying to find the implementation class of Stellar
functions. For example, "Where is the code that implements `PROFILE_GET`?"
Due to the way our `StellarFunction` annotation works (namespace="PROFILE",
function="GET"), there is not a simple way to just search for `PROFILE_GET`.
The search won't hit on anything.
That is why I though to add the phrase `PROFILE_GET` here, so the test class
can be more easily found.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311058843
##
File path:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/ProfilerClientFactory.java
##
@@ -0,0 +1,36 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+package org.apache.metron.profiler.client;
+
+import java.util.Map;
+
+/**
+ * Responsible for creating a {@link ProfilerClient}.
+ */
+public interface ProfilerClientFactory {
Review comment:
What are you suggesting? The "factory" in that example is called
`ExtractorCreator`. You think this should be `ProfilerClientCreator`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2 URL: https://github.com/apache/metron/pull/1458#discussion_r311057827 ## File path: metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/HBaseProfilerClient.java ## @@ -20,11 +20,11 @@ package org.apache.metron.profiler.client; Review comment: It's a client API that interacts with the Profiler. Personally, I don't see the name as being ambiguous. Even if it does make sense to change the name, I don't know that doing it in this PR makes sense. The name was not changed in this PR or on the HDP-3.1 feature branch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r311053614
##
File path:
metron-analytics/metron-profiler-storm/src/test/java/org/apache/metron/profiler/storm/integration/ProfilerIntegrationTest.java
##
@@ -503,21 +496,29 @@ public void setup() {
put(PROFILER_SALT_DIVISOR.getKey(), saltDivisor);
}};
+Context context = new Context.Builder()
+.with(Context.Capabilities.GLOBAL_CONFIG, () -> global)
+.build();
+
+// create the GET_PROFILE function
+GetProfile getProfileFunction = new GetProfile()
+.withProfilerClientFactory(new HBaseProfilerClientFactory(new
FakeHBaseClientFactory()));
Review comment:
Different tests needed different replacements. But in many cases, especially
the integration tests, using a `FakeHBaseClient` is the replacement for the
"mock Hbase table stuff".
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r307508158
##
File path:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/HBaseProfilerClientFactory.java
##
@@ -0,0 +1,98 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+package org.apache.metron.profiler.client;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.metron.hbase.client.HBaseClient;
+import org.apache.metron.hbase.client.HBaseClientFactory;
+import org.apache.metron.hbase.client.HBaseConnectionFactory;
+import org.apache.metron.hbase.client.HBaseTableClientFactory;
+import org.apache.metron.profiler.hbase.ColumnBuilder;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
+import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_COLUMN_FAMILY;
+import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE;
+import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_CONNECTION_FACTORY;
+import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+import static
org.apache.metron.profiler.client.stellar.Util.getPeriodDurationInMillis;
+
+/**
+ * Creates an {@link HBaseProfilerClient}.
+ */
+public class HBaseProfilerClientFactory implements ProfilerClientFactory {
+
+ private HBaseClientFactory hBaseClientFactory;
+
+ public HBaseProfilerClientFactory() {
+this(new HBaseTableClientFactory());
+ }
+
+ public HBaseProfilerClientFactory(HBaseClientFactory hBaseClientFactory) {
+this.hBaseClientFactory = hBaseClientFactory;
+ }
+
+ @Override
+ public HBaseProfilerClient create(Map globals) {
Review comment:
Agreed. After this PR and #1470 it should be easier to introduce some
improvements to `HBaseClient`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r307052275
##
File path:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/HBaseProfilerClientFactory.java
##
@@ -0,0 +1,98 @@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+package org.apache.metron.profiler.client;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.metron.hbase.client.HBaseClient;
+import org.apache.metron.hbase.client.HBaseClientFactory;
+import org.apache.metron.hbase.client.HBaseConnectionFactory;
+import org.apache.metron.hbase.client.HBaseTableClientFactory;
+import org.apache.metron.profiler.hbase.ColumnBuilder;
+import org.apache.metron.profiler.hbase.RowKeyBuilder;
+import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
+import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_COLUMN_FAMILY;
+import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE;
+import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_CONNECTION_FACTORY;
+import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+import static
org.apache.metron.profiler.client.stellar.Util.getPeriodDurationInMillis;
+
+/**
+ * Creates an {@link HBaseProfilerClient}.
+ */
+public class HBaseProfilerClientFactory implements ProfilerClientFactory {
+
+ private HBaseClientFactory hBaseClientFactory;
+
+ public HBaseProfilerClientFactory() {
+this(new HBaseTableClientFactory());
+ }
+
+ public HBaseProfilerClientFactory(HBaseClientFactory hBaseClientFactory) {
+this.hBaseClientFactory = hBaseClientFactory;
+ }
+
+ @Override
+ public HBaseProfilerClient create(Map globals) {
Review comment:
> I didn't catch this initially, but it looks like one subtlety here is
that your new connection interface is relevant on a per-table basis, so if you
need to modify 2 tables then you'd need to construct 2 connections to handle
that.
Yes, that is true. I am not aware of any places where this is an issue
currently. If this does come up, I think we can easily handle it by some new
implementations of `HBaseConnectionFactory` and/or `HBaseClientFactory`.
> I was thinking through this a bit and was considering whether it makes
sense to make the table operations more ad-hoc.
Keep in mind, I didn't invent the HBaseClient API as part of this HDP 3.1
upgrade effort. I am just reusing what we already have in
[master](https://github.com/apache/metron/blob/a2a46e6567c78e5066c0d3360d7838d1a5851fa7/metron-platform/metron-hbase/metron-hbase-common/src/main/java/org/apache/metron/hbase/client/HBaseClient.java#L145).
I'm not a big fan of the API itself, it is a bit clunky. It originally came
from the storm-hbase integration project and could certainly use some
improvements.
After we get all these Hbase upgrade changes in, we can certainly think
about improving the HBaseClient API. I don't think that doing it in the middle
of this upgrade makes the most sense.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2 URL: https://github.com/apache/metron/pull/1458#discussion_r306310081 ## File path: metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/HBaseProfilerClient.java ## @@ -44,27 +44,30 @@ /** * Used to access the profile data stored in HBase. */ - private HTableInterface table; + private HBaseClient hbaseClient; Review comment: The `HBaseProfilerClient` was updated to use our `HBaseClient` abstraction to interact with HBase. This also allows us to use the `FakeHBaseClient` when testing this class. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r306320231
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
##
@@ -20,437 +20,182 @@
package org.apache.metron.profiler.client.stellar;
-import org.apache.hadoop.hbase.client.HTableInterface;
-import org.apache.metron.hbase.mock.MockHBaseTableProvider;
import org.apache.metron.profiler.ProfileMeasurement;
-import org.apache.metron.profiler.client.ProfileWriter;
-import org.apache.metron.profiler.hbase.ColumnBuilder;
-import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
-import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
+import org.apache.metron.profiler.client.ProfilerClient;
import org.apache.metron.stellar.common.DefaultStellarStatefulExecutor;
import org.apache.metron.stellar.common.StellarStatefulExecutor;
import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.functions.resolver.FunctionResolver;
import org.apache.metron.stellar.dsl.functions.resolver.SimpleFunctionResolver;
-import
org.apache.metron.stellar.dsl.functions.resolver.SingletonFunctionResolver;
-import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
-import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.concurrent.TimeUnit;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_COLUMN_FAMILY;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_HBASE_TABLE_PROVIDER;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_PERIOD_UNITS;
-import static
org.apache.metron.profiler.client.stellar.ProfilerClientConfig.PROFILER_SALT_DIVISOR;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
/**
- * Tests the GetProfile class.
+ * Tests the 'PROFILE_GET' function in the {@link GetProfile} class.
*/
public class GetProfileTest {
- private static final long periodDuration = 15;
- private static final TimeUnit periodUnits = TimeUnit.MINUTES;
- private static final int saltDivisor = 1000;
- private static final String tableName = "profiler";
- private static final String columnFamily = "P";
private StellarStatefulExecutor executor;
- private Map state;
- private ProfileWriter profileWriter;
- // different values of period and salt divisor, used to test
config_overrides feature
- private static final long periodDuration2 = 1;
- private static final TimeUnit periodUnits2 = TimeUnit.HOURS;
- private static final int saltDivisor2 = 2050;
-
- private T run(String expression, Class clazz) {
-return executor.execute(expression, state, clazz);
+ private FunctionResolver functionResolver;
+ private Map globals;
+ private GetProfile function;
+ private ProfilerClient profilerClient;
+ private List results;
+ private ProfileMeasurement expected;
+
+ private List run(String expression) {
+return executor.execute(expression, new HashMap<>(), List.class);
}
- /**
- * This method sets up the configuration context for both writing profile
data
- * (using profileWriter to mock the complex process of what the Profiler
topology
- * actually does), and then reading that profile data (thereby testing the
PROFILE_GET
- * Stellar client implemented in GetProfile).
- *
- * It runs at @Before time, and sets testclass global variables used by the
writers and readers.
- * The various writers and readers are in each test case, not here.
- *
- * @return void
- */
@Before
public void setup() {
-state = new HashMap<>();
-final HTableInterface table = MockHBaseTableProvider.addToCache(tableName,
columnFamily);
-
-// used to write values to be read during testing
-long periodDurationMillis = TimeUnit.MINUTES.toMillis(15);
-RowKeyBuilder rowKeyBuilder = new SaltyRowKeyBuilder();
-ColumnBuilder columnBuilder = new ValueOnlyColumnBuilder(columnFamily);
-profileWriter = new ProfileWriter(rowKeyBuilder, columnBuilder, table,
periodDurationMillis);
-
-// global properties
-Map global = new HashMap() {{
- put(PROFILER_HBASE_TABLE.getKey(), tableName);
- put(PROFILER_COLUMN_FAMILY.getKey(), columnFamily);
-
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r306312837
##
File path:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java
##
@@ -102,94 +90,108 @@
returns="The selected profile measurements."
)
public class GetProfile implements StellarFunction {
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
/**
- * Cached client that can retrieve profile values.
+ * Allows the function to retrieve persisted {@link ProfileMeasurement}
values.
*/
- private ProfilerClient client;
+ private ProfilerClient profilerClient;
/**
- * Cached value of config map actually used to construct the previously
cached client.
+ * Creates the {@link ProfilerClient} used by this function.
*/
- private Map cachedConfigMap = new HashMap(6);
-
- private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private ProfilerClientFactory profilerClientFactory;
/**
- * Initialization. No longer need to do anything in initialization,
- * as all setup is done lazily and cached.
+ * Last known global configuration used to create the {@link
ProfilerClient}. If the
+ * global configuration changes, a new {@link ProfilerClient} needs to be
constructed.
*/
+ private Map lastKnownGlobals = new HashMap<>();
+
+ public GetProfile() {
+profilerClientFactory = new HBaseProfilerClientFactory();
+ }
+
@Override
public void initialize(Context context) {
+Map globals = getGlobals(context);
+profilerClient = profilerClientFactory.create(globals);
}
- /**
- * Is the function initialized?
- */
@Override
public boolean isInitialized() {
-return true;
+return profilerClient != null;
}
- /**
- * Apply the function.
- * @param args The function arguments.
- * @param context
- */
@Override
- public Object apply(List args, Context context) throws
ParseException {
+ public void close() throws IOException {
+if(profilerClient != null) {
+ profilerClient.close();
+}
+ }
+ @Override
+ public Object apply(List args, Context context) throws
ParseException {
+// required arguments
String profile = getArg(0, String.class, args);
String entity = getArg(1, String.class, args);
-Optional> periods = Optional.ofNullable(getArg(2,
List.class, args));
-//Optional arguments
Review comment:
Moved all of this logic around getting optional args into `getGroups` and
`getOverrides`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r306321103
##
File path:
metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/HBaseWriterFunction.java
##
@@ -117,8 +131,7 @@ public HBaseWriterFunction(Properties properties) {
if(measurements.size() > 0) {
// open an HBase connection
- Configuration config = HBaseConfiguration.create();
- try (LegacyHBaseClient client = new LegacyHBaseClient(tableProvider,
config, tableName)) {
+ try (HBaseClient client = hBaseClientFactory.create(connectionFactory,
HBaseConfiguration.create(), tableName)) {
Review comment:
Make the Batch Profiler write to Hbase using the new `HbaseClient`
abstraction.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r306311766
##
File path:
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/ProfilerClient.java
##
@@ -23,13 +23,14 @@
import org.apache.metron.profiler.ProfileMeasurement;
import org.apache.metron.profiler.ProfilePeriod;
+import java.io.Closeable;
import java.util.List;
import java.util.Optional;
/**
* An interface for a client capable of retrieving the profile data that has
been persisted by the Profiler.
*/
-public interface ProfilerClient {
+public interface ProfilerClient extends Closeable {
Review comment:
Needs to be closable so that the underlying Hbase connections can be
cleaned-up.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r306318215
##
File path:
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/HBaseProfilerClientTest.java
##
@@ -20,184 +20,187 @@
package org.apache.metron.profiler.client;
-import org.apache.metron.hbase.mock.MockHTable;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.common.utils.SerDeUtils;
+import org.apache.metron.hbase.client.HBaseTableClient;
+import org.apache.metron.hbase.client.HBaseClient;
import org.apache.metron.profiler.ProfileMeasurement;
+import org.apache.metron.profiler.ProfilePeriod;
import org.apache.metron.profiler.hbase.ColumnBuilder;
import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.profiler.hbase.SaltyRowKeyBuilder;
-import org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder;
-import org.apache.metron.stellar.common.DefaultStellarStatefulExecutor;
-import org.apache.metron.stellar.common.StellarStatefulExecutor;
-import org.junit.After;
import org.junit.Before;
import org.junit.Test;
+import org.mockito.ArgumentCaptor;
+import java.io.IOException;
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.TimeUnit;
import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+import static org.mockito.Mockito.any;
+import static org.mockito.Mockito.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
/**
- * Tests the HBaseProfilerClient.
- *
- * The naming used in this test attempts to be as similar to how the 'groupBy'
- * functionality might be used 'in the wild'. This test involves reading and
- * writing two separate groups originating from the same Profile and Entity.
- * There is a 'weekdays' group which contains all measurements taken on
weekdays.
- * There is also a 'weekend' group which contains all measurements taken on
weekends.
+ * Tests the {@link HBaseProfilerClient}.
*/
public class HBaseProfilerClientTest {
-
private static final String tableName = "profiler";
private static final String columnFamily = "P";
+ private static final byte[] columnFamilyB = Bytes.toBytes(columnFamily);
+ private static final byte[] columnQualifier = Bytes.toBytes("column");
private static final long periodDuration = 15;
private static final TimeUnit periodUnits = TimeUnit.MINUTES;
private static final int periodsPerHour = 4;
-
- private HBaseProfilerClient client;
- private StellarStatefulExecutor executor;
- private MockHTable table;
- private ProfileWriter profileWriter;
+ private static final byte[] expectedRowKey = Bytes.toBytes("some-row-key");
+ private static final String profileName = "profile1";
+ private static final String entityName = "entity1";
+ private static final int profileValue = 1231121;
+ private static final byte[] profileValueB = SerDeUtils.toBytes(profileValue);
+ private long periodDurationMillis = periodUnits.toMillis(periodDuration);
+
+ private HBaseClient hbaseClient;
+ private HBaseProfilerClient profilerClient;
+ private ProfileMeasurement expected;
+ private RowKeyBuilder rowKeyBuilder;
+ private ColumnBuilder columnBuilder;
+ private Result expectedResult;
+ private Result emptyResult;
@Before
- public void setup() throws Exception {
-table = new MockHTable(tableName, columnFamily);
-executor = new DefaultStellarStatefulExecutor();
-
-// writes values to be read during testing
-long periodDurationMillis = periodUnits.toMillis(periodDuration);
-RowKeyBuilder rowKeyBuilder = new SaltyRowKeyBuilder();
-ColumnBuilder columnBuilder = new ValueOnlyColumnBuilder(columnFamily);
-profileWriter = new ProfileWriter(rowKeyBuilder, columnBuilder, table,
periodDurationMillis);
-
-client = new HBaseProfilerClient(table, rowKeyBuilder, columnBuilder,
periodDurationMillis);
+ public void setup() {
+// create a profile measurement used in the tests
+expected = new ProfileMeasurement()
+.withProfileName(profileName)
+.withEntity(entityName)
+.withPeriod(System.currentTimeMillis(), 5, TimeUnit.MINUTES)
+.withProfileValue(profileValue);
+
+// mock row key builder needs to return a row key for the profile
measurement used in the tests
+rowKeyBuilder = mock(RowKeyBuilder.class);
+when(rowKeyBuilder.rowKey(any())).thenReturn(expectedRowKey);
+
+// mock column builder - column family/qualifier comes from the column
builder
+columnBuilder = mock(ColumnBuilder.class);
+
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r306327005
##
File path:
metron-analytics/metron-profiler-storm/src/test/java/org/apache/metron/profiler/storm/integration/ProfilerIntegrationTest.java
##
@@ -503,21 +496,29 @@ public void setup() {
put(PROFILER_SALT_DIVISOR.getKey(), saltDivisor);
}};
+Context context = new Context.Builder()
+.with(Context.Capabilities.GLOBAL_CONFIG, () -> global)
+.build();
+
+// create the GET_PROFILE function
+GetProfile getProfileFunction = new GetProfile()
+.withProfilerClientFactory(new HBaseProfilerClientFactory(new
FakeHBaseClientFactory()));
Review comment:
We need both the Stellar functions and the topology running in Storm to
"see" the same "HBase data" even though we can't run HBase with this
integration test. Both the topology and the functions see the same static data
presented by the `FakeHBaseClient`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade Profiler for HBase 2.0.2
nickwallen commented on a change in pull request #1458: METRON-2177 Upgrade
Profiler for HBase 2.0.2
URL: https://github.com/apache/metron/pull/1458#discussion_r306327795
##
File path:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/resolver/BaseFunctionResolver.java
##
@@ -242,4 +242,18 @@ public static StellarFunction createFunction(Class cl
return null;
}
}
+
+ @Override
+ public BaseFunctionResolver withInstance(StellarFunction function) {
+// perform function resolution on the instance that was passed in
Review comment:
Allows us to inject a `StellarFunction` instance into the set of resolved
functions.
This is what solves the problem of how do we ensure that Stellar functions
and code running in a Storm topology "see" the same underlying HBase data,
without being able to run HBase and no longer having the mock HBase logic that
existed previously.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
