[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/795


---

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/795#discussion_r146031435
  
--- Diff: 
metron-platform/metron-zookeeper/src/main/java/org/apache/metron/zookeeper/ZKCache.java
 ---
@@ -0,0 +1,141 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.zookeeper;
+
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+public class ZKCache implements AutoCloseable{
+  private static final Logger LOG =  
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+  public static final int DEFAULT_CLIENT_SLEEP_MS = 1000;
+  public static final int DEFAULT_MAX_RETRIES = 3;
+
+
+
+
+  public static class Builder {
+private Optional client = Optional.empty();
+private boolean ownClient = false;
+private List listener = new ArrayList<>();
+private String zkRoot;
+
+public Builder() { }
+
+public Builder withClient(CuratorFramework client) {
+  this.client = Optional.ofNullable(client);
+  ownClient = false;
+  return this;
+}
+
+public Builder withClient(String zookeeperUrl) {
+  this.client = Optional.ofNullable(createClient(zookeeperUrl, 
Optional.empty()));
+  ownClient = true;
+  return this;
+}
+
+public Builder withClient(String zookeeperUrl, RetryPolicy 
retryPolicy) {
+  this.client = Optional.ofNullable(createClient(zookeeperUrl, 
Optional.ofNullable(retryPolicy)));
+  ownClient = true;
+  return this;
+}
+
+public Builder withListener(TreeCacheListener listener) {
+  this.listener.add(listener);
+  return this;
+}
+
+public Builder withRoot(String zkRoot) {
+  this.zkRoot = zkRoot;
+  return this;
+}
+
+public ZKCache build() {
+  if(!client.isPresent()) {
+throw new IllegalArgumentException("Zookeeper client must be 
specified.");
+  }
+  if(listener.isEmpty()) {
+LOG.warn("Zookeeper listener is null or empty, which is very 
likely an error.");
+  }
+  if(zkRoot == null) {
+throw new IllegalArgumentException("Zookeeper root must not be 
null.");
+  }
+  return new ZKCache(client.get(), listener, zkRoot, ownClient);
+}
+
+  }
+
+  private CuratorFramework client;
+  private List listeners;
+  private TreeCache cache;
+  private String zkRoot;
+  private boolean ownClient = false;
+
+  private ZKCache(CuratorFramework client, List 
listeners, String zkRoot, boolean ownClient) {
--- End diff --

Sure thing.


---

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/795#discussion_r146021528
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/zookeeper/configurations/ProfilerUpdater.java
 ---
@@ -0,0 +1,114 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.zookeeper.configurations;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.ConfigurationType;
+import org.apache.metron.common.configuration.ConfigurationsUtils;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import 
org.apache.metron.common.configuration.profiler.ProfilerConfigurations;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.zookeeper.KeeperException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+public class ProfilerUpdater extends 
ConfigurationsUpdater {
+  public ProfilerUpdater(Reloadable reloadable, 
Supplier configSupplier) {
+super(reloadable, configSupplier);
+  }
+
+  @Override
+  public Class getConfigurationClass() {
+return ProfilerConfigurations.class;
+  }
+
+  private ProfilerConfig readFromZookeeper(CuratorFramework client) throws 
Exception {
+byte[] raw = client.getData().forPath(PROFILER.getZookeeperRoot());
+return JSONUtils.INSTANCE.load(new ByteArrayInputStream(raw), 
ProfilerConfig.class);
+  }
+
+  @Override
+  public void forceUpdate(CuratorFramework client) {
+try {
+  ConfigurationsUtils.updateConfigsFromZookeeper(getConfigurations(), 
client);
+}
+catch (KeeperException.NoNodeException nne) {
+  LOG.warn("No current global configs in zookeeper, but the cache 
should load lazily...");
+}
+catch(Exception e) {
+  LOG.warn("Unable to load global configs from zookeeper, but the 
cache should load lazily...", e);
+}
+try {
+  ProfilerConfig config = readFromZookeeper(client);
+  if(config != null) {
+getConfigurations().updateProfilerConfig(config);
+  }
--- End diff --

@cestella  Saw your response only after my last post.  Yes, makes complete 
sense.  Thanks


---

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/795#discussion_r146021013
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/zookeeper/configurations/ProfilerUpdater.java
 ---
@@ -0,0 +1,114 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.zookeeper.configurations;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.ConfigurationType;
+import org.apache.metron.common.configuration.ConfigurationsUtils;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import 
org.apache.metron.common.configuration.profiler.ProfilerConfigurations;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.zookeeper.KeeperException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+public class ProfilerUpdater extends 
ConfigurationsUpdater {
+  public ProfilerUpdater(Reloadable reloadable, 
Supplier configSupplier) {
+super(reloadable, configSupplier);
+  }
+
+  @Override
+  public Class getConfigurationClass() {
+return ProfilerConfigurations.class;
+  }
+
+  private ProfilerConfig readFromZookeeper(CuratorFramework client) throws 
Exception {
+byte[] raw = client.getData().forPath(PROFILER.getZookeeperRoot());
+return JSONUtils.INSTANCE.load(new ByteArrayInputStream(raw), 
ProfilerConfig.class);
+  }
+
+  @Override
+  public void forceUpdate(CuratorFramework client) {
+try {
+  ConfigurationsUtils.updateConfigsFromZookeeper(getConfigurations(), 
client);
+}
+catch (KeeperException.NoNodeException nne) {
+  LOG.warn("No current global configs in zookeeper, but the cache 
should load lazily...");
+}
+catch(Exception e) {
+  LOG.warn("Unable to load global configs from zookeeper, but the 
cache should load lazily...", e);
+}
+try {
+  ProfilerConfig config = readFromZookeeper(client);
+  if(config != null) {
+getConfigurations().updateProfilerConfig(config);
+  }
--- End diff --

So to summarize, that was me answering my own question.  Ignore this. :)


---

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/795#discussion_r146020850
  
--- Diff: 
metron-platform/metron-zookeeper/src/main/java/org/apache/metron/zookeeper/ZKCache.java
 ---
@@ -0,0 +1,141 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.zookeeper;
+
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+public class ZKCache implements AutoCloseable{
+  private static final Logger LOG =  
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+  public static final int DEFAULT_CLIENT_SLEEP_MS = 1000;
+  public static final int DEFAULT_MAX_RETRIES = 3;
+
+
+
+
+  public static class Builder {
+private Optional client = Optional.empty();
+private boolean ownClient = false;
+private List listener = new ArrayList<>();
+private String zkRoot;
+
+public Builder() { }
+
+public Builder withClient(CuratorFramework client) {
+  this.client = Optional.ofNullable(client);
+  ownClient = false;
+  return this;
+}
+
+public Builder withClient(String zookeeperUrl) {
+  this.client = Optional.ofNullable(createClient(zookeeperUrl, 
Optional.empty()));
+  ownClient = true;
+  return this;
+}
+
+public Builder withClient(String zookeeperUrl, RetryPolicy 
retryPolicy) {
+  this.client = Optional.ofNullable(createClient(zookeeperUrl, 
Optional.ofNullable(retryPolicy)));
+  ownClient = true;
+  return this;
+}
+
+public Builder withListener(TreeCacheListener listener) {
+  this.listener.add(listener);
+  return this;
+}
+
+public Builder withRoot(String zkRoot) {
+  this.zkRoot = zkRoot;
+  return this;
+}
+
+public ZKCache build() {
+  if(!client.isPresent()) {
+throw new IllegalArgumentException("Zookeeper client must be 
specified.");
+  }
+  if(listener.isEmpty()) {
+LOG.warn("Zookeeper listener is null or empty, which is very 
likely an error.");
+  }
+  if(zkRoot == null) {
+throw new IllegalArgumentException("Zookeeper root must not be 
null.");
+  }
+  return new ZKCache(client.get(), listener, zkRoot, ownClient);
+}
+
+  }
+
+  private CuratorFramework client;
+  private List listeners;
+  private TreeCache cache;
+  private String zkRoot;
+  private boolean ownClient = false;
+
+  private ZKCache(CuratorFramework client, List 
listeners, String zkRoot, boolean ownClient) {
--- End diff --

That helps a lot for ZkCache.  

Can you do the same for the other new classes?  I think that includes 
ConfigurationsCache, ConfigurationsUpdater and SimpleEventListener, if I am not 
mistaken.


---

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/795#discussion_r146020712
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/zookeeper/configurations/IndexingUpdater.java
 ---
@@ -0,0 +1,88 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.zookeeper.configurations;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.ConfigurationType;
+import org.apache.metron.common.configuration.ConfigurationsUtils;
+import org.apache.metron.common.configuration.IndexingConfigurations;
+import org.apache.zookeeper.KeeperException;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+public class IndexingUpdater extends 
ConfigurationsUpdater {
+  public IndexingUpdater(Reloadable reloadable, 
Supplier configSupplier) {
+super(reloadable, configSupplier);
+  }
+
+  @Override
+  public Class getConfigurationClass() {
+return IndexingConfigurations.class;
+  }
+
+  @Override
+  public void forceUpdate(CuratorFramework client) {
+try {
+  
ConfigurationsUtils.updateSensorIndexingConfigsFromZookeeper(getConfigurations(),
 client);
+}
+catch (KeeperException.NoNodeException nne) {
+  LOG.warn("No current indexing configs in zookeeper, but the cache 
should load lazily...");
+}
+catch (Exception e) {
+  LOG.warn("Unable to load indexing configs from zookeeper, but the 
cache should load lazily...", e);
+}
+  }
+
+  @Override
+  public IndexingConfigurations defaultConfigurations() {
+return new IndexingConfigurations();
+  }
+
+  @Override
+  public void delete(CuratorFramework client, String path, byte[] data) 
throws IOException {
+String name = path.substring(path.lastIndexOf("/") + 1);
+if (path.startsWith(ConfigurationType.INDEXING.getZookeeperRoot())) {
--- End diff --

Yeah, I'll do that.  It's a good catch, thanks @nickwallen 


---

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/795#discussion_r146020428
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/zookeeper/configurations/ProfilerUpdater.java
 ---
@@ -0,0 +1,114 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.zookeeper.configurations;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.ConfigurationType;
+import org.apache.metron.common.configuration.ConfigurationsUtils;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import 
org.apache.metron.common.configuration.profiler.ProfilerConfigurations;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.zookeeper.KeeperException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+public class ProfilerUpdater extends 
ConfigurationsUpdater {
+  public ProfilerUpdater(Reloadable reloadable, 
Supplier configSupplier) {
+super(reloadable, configSupplier);
+  }
+
+  @Override
+  public Class getConfigurationClass() {
+return ProfilerConfigurations.class;
+  }
+
+  private ProfilerConfig readFromZookeeper(CuratorFramework client) throws 
Exception {
+byte[] raw = client.getData().forPath(PROFILER.getZookeeperRoot());
+return JSONUtils.INSTANCE.load(new ByteArrayInputStream(raw), 
ProfilerConfig.class);
+  }
+
+  @Override
+  public void forceUpdate(CuratorFramework client) {
+try {
+  ConfigurationsUtils.updateConfigsFromZookeeper(getConfigurations(), 
client);
+}
+catch (KeeperException.NoNodeException nne) {
+  LOG.warn("No current global configs in zookeeper, but the cache 
should load lazily...");
+}
+catch(Exception e) {
+  LOG.warn("Unable to load global configs from zookeeper, but the 
cache should load lazily...", e);
+}
+try {
+  ProfilerConfig config = readFromZookeeper(client);
+  if(config != null) {
+getConfigurations().updateProfilerConfig(config);
+  }
--- End diff --

This is because the Profiler doesn't have a call in ConfigurationsUtils, 
unlike the rest of our fundamental types (e.g. 
`update{Parser,Enrichment,etc}ConfigsFromZookeeper`).  When calling those, an 
update to the global config is called and then the configs are updated.  This 
is done here and it's why it's a bit different.  I thought about moving this 
abstraction to `ConfigurationsUtils`, but I felt that class was already so 
onerous and needed to be fundamentally refactored so deeply that I didn't have 
the heart to create a `updateProfilerConfigsFromZookeeper` method.


---

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/795#discussion_r146019541
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/zookeeper/configurations/ProfilerUpdater.java
 ---
@@ -0,0 +1,114 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.zookeeper.configurations;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.ConfigurationType;
+import org.apache.metron.common.configuration.ConfigurationsUtils;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import 
org.apache.metron.common.configuration.profiler.ProfilerConfigurations;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.zookeeper.KeeperException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+public class ProfilerUpdater extends 
ConfigurationsUpdater {
+  public ProfilerUpdater(Reloadable reloadable, 
Supplier configSupplier) {
+super(reloadable, configSupplier);
+  }
+
+  @Override
+  public Class getConfigurationClass() {
+return ProfilerConfigurations.class;
+  }
+
+  private ProfilerConfig readFromZookeeper(CuratorFramework client) throws 
Exception {
+byte[] raw = client.getData().forPath(PROFILER.getZookeeperRoot());
+return JSONUtils.INSTANCE.load(new ByteArrayInputStream(raw), 
ProfilerConfig.class);
+  }
+
+  @Override
+  public void forceUpdate(CuratorFramework client) {
+try {
+  ConfigurationsUtils.updateConfigsFromZookeeper(getConfigurations(), 
client);
+}
+catch (KeeperException.NoNodeException nne) {
+  LOG.warn("No current global configs in zookeeper, but the cache 
should load lazily...");
+}
+catch(Exception e) {
+  LOG.warn("Unable to load global configs from zookeeper, but the 
cache should load lazily...", e);
+}
+try {
+  ProfilerConfig config = readFromZookeeper(client);
+  if(config != null) {
+getConfigurations().updateProfilerConfig(config);
+  }
--- End diff --

Hmm, looks-like just a difference in what is offered in 
`ConfigurationUtils`.  Seems necessary unless we were to unpack some of the 
logic from `ConfigurationUtils`.  But diving into `ConfigurationUtils` seems 
more effort than it is worth for this change.


---

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/795#discussion_r146018459
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/zookeeper/configurations/ProfilerUpdater.java
 ---
@@ -0,0 +1,114 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.zookeeper.configurations;
+
+import static 
org.apache.metron.common.configuration.ConfigurationType.PROFILER;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.ConfigurationType;
+import org.apache.metron.common.configuration.ConfigurationsUtils;
+import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import 
org.apache.metron.common.configuration.profiler.ProfilerConfigurations;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.zookeeper.KeeperException;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.function.Supplier;
+
+public class ProfilerUpdater extends 
ConfigurationsUpdater {
+  public ProfilerUpdater(Reloadable reloadable, 
Supplier configSupplier) {
+super(reloadable, configSupplier);
+  }
+
+  @Override
+  public Class getConfigurationClass() {
+return ProfilerConfigurations.class;
+  }
+
+  private ProfilerConfig readFromZookeeper(CuratorFramework client) throws 
Exception {
+byte[] raw = client.getData().forPath(PROFILER.getZookeeperRoot());
+return JSONUtils.INSTANCE.load(new ByteArrayInputStream(raw), 
ProfilerConfig.class);
+  }
+
+  @Override
+  public void forceUpdate(CuratorFramework client) {
+try {
+  ConfigurationsUtils.updateConfigsFromZookeeper(getConfigurations(), 
client);
+}
+catch (KeeperException.NoNodeException nne) {
+  LOG.warn("No current global configs in zookeeper, but the cache 
should load lazily...");
+}
+catch(Exception e) {
+  LOG.warn("Unable to load global configs from zookeeper, but the 
cache should load lazily...", e);
+}
+try {
+  ProfilerConfig config = readFromZookeeper(client);
+  if(config != null) {
+getConfigurations().updateProfilerConfig(config);
+  }
--- End diff --

The `ProfilerUpdater` is the only `ConfigurationsUpdater` implementation 
that has this additional logic in lines 61-73.  Looks fishy.  Why is it special?


---

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/795#discussion_r146012465
  
--- Diff: 
metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/StormControllerIntegrationTest.java
 ---
@@ -179,6 +181,8 @@ public void test() throws Exception {
 
sensorParserConfig.setParserClassName("org.apache.metron.parsers.bro.BasicBroParser");
 sensorParserConfig.setSensorTopic("broTest");
 sensorParserConfigService.save(sensorParserConfig);
+//we must wait for the config to find its way into the config.
+Thread.sleep(500);
--- End diff --

Yep, will change.


---

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/795#discussion_r146010894
  
--- Diff: 
metron-platform/metron-zookeeper/src/main/java/org/apache/metron/zookeeper/ZKCache.java
 ---
@@ -0,0 +1,141 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.zookeeper;
+
+import org.apache.curator.RetryPolicy;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.recipes.cache.TreeCache;
+import org.apache.curator.framework.recipes.cache.TreeCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+
+public class ZKCache implements AutoCloseable{
+  private static final Logger LOG =  
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+  public static final int DEFAULT_CLIENT_SLEEP_MS = 1000;
+  public static final int DEFAULT_MAX_RETRIES = 3;
+
+
+
+
+  public static class Builder {
+private Optional client = Optional.empty();
+private boolean ownClient = false;
+private List listener = new ArrayList<>();
+private String zkRoot;
+
+public Builder() { }
+
+public Builder withClient(CuratorFramework client) {
+  this.client = Optional.ofNullable(client);
+  ownClient = false;
+  return this;
+}
+
+public Builder withClient(String zookeeperUrl) {
+  this.client = Optional.ofNullable(createClient(zookeeperUrl, 
Optional.empty()));
+  ownClient = true;
+  return this;
+}
+
+public Builder withClient(String zookeeperUrl, RetryPolicy 
retryPolicy) {
+  this.client = Optional.ofNullable(createClient(zookeeperUrl, 
Optional.ofNullable(retryPolicy)));
+  ownClient = true;
+  return this;
+}
+
+public Builder withListener(TreeCacheListener listener) {
+  this.listener.add(listener);
+  return this;
+}
+
+public Builder withRoot(String zkRoot) {
+  this.zkRoot = zkRoot;
+  return this;
+}
+
+public ZKCache build() {
+  if(!client.isPresent()) {
+throw new IllegalArgumentException("Zookeeper client must be 
specified.");
+  }
+  if(listener.isEmpty()) {
+LOG.warn("Zookeeper listener is null or empty, which is very 
likely an error.");
+  }
+  if(zkRoot == null) {
+throw new IllegalArgumentException("Zookeeper root must not be 
null.");
+  }
+  return new ZKCache(client.get(), listener, zkRoot, ownClient);
+}
+
+  }
+
+  private CuratorFramework client;
+  private List listeners;
+  private TreeCache cache;
+  private String zkRoot;
+  private boolean ownClient = false;
+
+  private ZKCache(CuratorFramework client, List 
listeners, String zkRoot, boolean ownClient) {
--- End diff --

I think the PR is looking great.  It cleans a fair bit up quite nicely.  

Can you add javadoc in the appropriate places?  For instance, what is 
'ownClient' and when/how do I use it?  A few javadocs would make it so I don't 
have to trace through the code to find out.


---

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/795#discussion_r145992573
  
--- Diff: 
metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/StormControllerIntegrationTest.java
 ---
@@ -179,6 +181,8 @@ public void test() throws Exception {
 
sensorParserConfig.setParserClassName("org.apache.metron.parsers.bro.BasicBroParser");
 sensorParserConfig.setSensorTopic("broTest");
 sensorParserConfigService.save(sensorParserConfig);
+//we must wait for the config to find its way into the config.
+Thread.sleep(500);
--- End diff --

Would it make sense to use assertEventually here instead of Thread.sleep?


---

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[cestella]

GitHub user cestella reopened a pull request:

https://github.com/apache/metron/pull/795

METRON-1241: Enable the REST API to use a cache for the zookeeper config 
similar to the Bolts

## Contributor Comments
Currently, our bolts use a TreeCache to capture and update internal state. 
The REST API, on the other hand, polls zookeeper every time a request is made 
to retrieve a configuration, a poll model. Rather than do this, it would be 
better to share an abstraction between the two pieces of infrastructure to 
enable the REST API to be backed by a TreeCache, which will prevent zookeeper 
from being abused and migrate it to a push model.

Testing instruction to come.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron zookeeper_refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/795.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #795


commit 0570ba18d773688bfc2d71e2f081155583c21a2e
Author: cstella 
Date:   2017-10-09T19:03:32Z

Abstracting zookeeper substantially.

commit e59a82266c7cefb2a4fe6c033e9b14c2fe7c001b
Author: cstella 
Date:   2017-10-09T20:10:57Z

Updating tests to use the new API.

commit 8d64f05f33bb248ae3306cb580ac589400887b81
Author: cstella 
Date:   2017-10-09T20:15:51Z

Missed a couple of tests.

commit a195ae51e3c5a04079c2b401ae325bcf4e46c05e
Author: cstella 
Date:   2017-10-09T20:32:41Z

moved the cache into commons.

commit 0932c30e9af4b75285c7003e722888e4a5675f9f
Author: cstella 
Date:   2017-10-09T21:15:20Z

Updating test.

commit 7c05716ceb6db505ec323f7b04716df07399a05c
Author: cstella 
Date:   2017-10-09T23:41:44Z

Refactored tests to mock the configurations cache

commit 8a295a271fe94995283e02b2b291f50ec2db30ef
Author: cstella 
Date:   2017-10-10T00:13:41Z

fixing tests.

commit dd41e6ed9a28a3b30102aad76f5396c51cf540f4
Author: cstella 

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[cestella]

Github user cestella closed the pull request at:

https://github.com/apache/metron/pull/795


---

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[cestella]

GitHub user cestella reopened a pull request:

https://github.com/apache/metron/pull/795

METRON-1241: Enable the REST API to use a cache for the zookeeper config 
similar to the Bolts

## Contributor Comments
Currently, our bolts use a TreeCache to capture and update internal state. 
The REST API, on the other hand, polls zookeeper every time a request is made 
to retrieve a configuration, a poll model. Rather than do this, it would be 
better to share an abstraction between the two pieces of infrastructure to 
enable the REST API to be backed by a TreeCache, which will prevent zookeeper 
from being abused and migrate it to a push model.

Testing instruction to come.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron zookeeper_refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/795.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #795


commit 0570ba18d773688bfc2d71e2f081155583c21a2e
Author: cstella 
Date:   2017-10-09T19:03:32Z

Abstracting zookeeper substantially.

commit e59a82266c7cefb2a4fe6c033e9b14c2fe7c001b
Author: cstella 
Date:   2017-10-09T20:10:57Z

Updating tests to use the new API.

commit 8d64f05f33bb248ae3306cb580ac589400887b81
Author: cstella 
Date:   2017-10-09T20:15:51Z

Missed a couple of tests.

commit a195ae51e3c5a04079c2b401ae325bcf4e46c05e
Author: cstella 
Date:   2017-10-09T20:32:41Z

moved the cache into commons.

commit 0932c30e9af4b75285c7003e722888e4a5675f9f
Author: cstella 
Date:   2017-10-09T21:15:20Z

Updating test.

commit 7c05716ceb6db505ec323f7b04716df07399a05c
Author: cstella 
Date:   2017-10-09T23:41:44Z

Refactored tests to mock the configurations cache

commit 8a295a271fe94995283e02b2b291f50ec2db30ef
Author: cstella 
Date:   2017-10-10T00:13:41Z

fixing tests.

commit dd41e6ed9a28a3b30102aad76f5396c51cf540f4
Author: cstella 

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[cestella]

Github user cestella closed the pull request at:

https://github.com/apache/metron/pull/795


---

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[cestella]

Github user cestella closed the pull request at:

https://github.com/apache/metron/pull/795


---

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[cestella]

GitHub user cestella reopened a pull request:

https://github.com/apache/metron/pull/795

METRON-1241: Enable the REST API to use a cache for the zookeeper config 
similar to the Bolts

## Contributor Comments
Currently, our bolts use a TreeCache to capture and update internal state. 
The REST API, on the other hand, polls zookeeper every time a request is made 
to retrieve a configuration, a poll model. Rather than do this, it would be 
better to share an abstraction between the two pieces of infrastructure to 
enable the REST API to be backed by a TreeCache, which will prevent zookeeper 
from being abused and migrate it to a push model.

Testing instruction to come.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron zookeeper_refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/795.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #795


commit 0570ba18d773688bfc2d71e2f081155583c21a2e
Author: cstella 
Date:   2017-10-09T19:03:32Z

Abstracting zookeeper substantially.

commit e59a82266c7cefb2a4fe6c033e9b14c2fe7c001b
Author: cstella 
Date:   2017-10-09T20:10:57Z

Updating tests to use the new API.

commit 8d64f05f33bb248ae3306cb580ac589400887b81
Author: cstella 
Date:   2017-10-09T20:15:51Z

Missed a couple of tests.

commit a195ae51e3c5a04079c2b401ae325bcf4e46c05e
Author: cstella 
Date:   2017-10-09T20:32:41Z

moved the cache into commons.

commit 0932c30e9af4b75285c7003e722888e4a5675f9f
Author: cstella 
Date:   2017-10-09T21:15:20Z

Updating test.

commit 7c05716ceb6db505ec323f7b04716df07399a05c
Author: cstella 
Date:   2017-10-09T23:41:44Z

Refactored tests to mock the configurations cache

commit 8a295a271fe94995283e02b2b291f50ec2db30ef
Author: cstella 
Date:   2017-10-10T00:13:41Z

fixing tests.

commit dd41e6ed9a28a3b30102aad76f5396c51cf540f4
Author: cstella 

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[cestella]

Github user cestella closed the pull request at:

https://github.com/apache/metron/pull/795


---

[GitHub] metron pull request #795: METRON-1241: Enable the REST API to use a cache fo...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/795

METRON-1241: Enable the REST API to use a cache for the zookeeper config 
similar to the Bolts

## Contributor Comments
Currently, our bolts use a TreeCache to capture and update internal state. 
The REST API, on the other hand, polls zookeeper every time a request is made 
to retrieve a configuration, a poll model. Rather than do this, it would be 
better to share an abstraction between the two pieces of infrastructure to 
enable the REST API to be backed by a TreeCache, which will prevent zookeeper 
from being abused and migrate it to a push model.

Testing instruction to come.


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron zookeeper_refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/795.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #795


commit 0570ba18d773688bfc2d71e2f081155583c21a2e
Author: cstella 
Date:   2017-10-09T19:03:32Z

Abstracting zookeeper substantially.

commit e59a82266c7cefb2a4fe6c033e9b14c2fe7c001b
Author: cstella 
Date:   2017-10-09T20:10:57Z

Updating tests to use the new API.

commit 8d64f05f33bb248ae3306cb580ac589400887b81
Author: cstella 
Date:   2017-10-09T20:15:51Z

Missed a couple of tests.

commit a195ae51e3c5a04079c2b401ae325bcf4e46c05e
Author: cstella 
Date:   2017-10-09T20:32:41Z

moved the cache into commons.

commit 0932c30e9af4b75285c7003e722888e4a5675f9f
Author: cstella 
Date:   2017-10-09T21:15:20Z

Updating test.

commit 7c05716ceb6db505ec323f7b04716df07399a05c
Author: cstella 
Date:   2017-10-09T23:41:44Z

Refactored tests to mock the configurations cache

commit 8a295a271fe94995283e02b2b291f50ec2db30ef
Author: cstella 
Date:   2017-10-10T00:13:41Z

fixing tests.

commit dd41e6ed9a28a3b30102aad76f5396c51cf540f4
Author: cstella