Re: [GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[Matt Foley]

Eesh, sorry, please ignore.  This was incorrect, due to reading two diff blocks 
as contiguous. 

On 8/16/17, 9:58 PM, "mattf-horton"  wrote:

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133623540
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -119,8 +120,10 @@ public static void init(final FileSystemManager 
fileSystemManager, final List

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133623540
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -119,8 +120,10 @@ public static void init(final FileSystemManager 
fileSystemManager, final List

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133619506
  
--- Diff: 
bundles-lib/src/test/java/org/apache/metron/bundles/BundleThreadContextClassLoaderTest.java
 ---
@@ -27,11 +27,21 @@
 import org.apache.metron.bundles.bundle.Bundle;
 import org.apache.metron.bundles.util.BundleProperties;
 import org.apache.metron.bundles.util.FileSystemManagerFactory;
+import org.junit.After;
 import org.junit.AfterClass;
 import org.junit.Test;
 
 public class BundleThreadContextClassLoaderTest {
+  @AfterClass
+  public static void after() {
+BundleClassLoaders.reset();
+  }
--- End diff --

I hate this static stuff. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133619456
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -95,19 +91,11 @@ public static BundleClassLoaders getInstance() {
*/
   public static void reset() {
--- End diff --

ok



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133619481
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -103,61 +104,52 @@ private ExtensionManager(){}
   /**
* @return The singleton instance of the ExtensionManager
*/
-  public static ExtensionManager getInstance() {
-ExtensionManager result = extensionManager;
-if (result == null) {
-  synchronized (ExtensionManager.class) {
-result = extensionManager;
-if (result == null) {
-  extensionManager = new ExtensionManager();
-  result = extensionManager;
-}
+  public static ExtensionManager getInstance() throws 
NotInitializedException {
+synchronized (ExtensionManager.class) {
+  if (extensionManager == null) {
+throw new NotInitializedException("ExtensionManager not 
initialized");
   }
+  return extensionManager;
 }
-return result;
   }
 
   /**
* Uninitializes the ExtensionManager.
* TESTING ONLY
*/
+  @VisibleForTesting
   public static void reset() {
 synchronized (ExtensionManager.class) {
-  getInstance().forgetExtensions();
+  initContext = null;
   extensionManager = null;
 }
   }
 
-  private void forgetExtensions() {
-  initContext = null;
-  }
-
-
   /**
* Loads all extension class types that can be found on the bootstrap 
classloader and by creating
* classloaders for all BUNDLES found within the classpath.
*
* @param bundles the bundles to scan through in search of extensions
*/
-  public void init(final List classes, final Bundle systemBundle, 
final Set bundles)
+  public static void init(final List classes, final Bundle 
systemBundle, final Set bundles)
   throws NotInitializedException {
 
-InitContext ic = initContext;
-if (ic == null) {
-  synchronized (this) {
-ic = initContext;
-if (ic == null) {
-  initContext = discoverExtensions(classes, systemBundle, bundles);
-  ic = initContext;
-}
+if (systemBundle == null) {
+  throw new IllegalArgumentException("systemBundle is required");
+}
+
+synchronized (ExtensionManager.class) {
+  if (extensionManager != null) {
+throw new IllegalStateException("ExtensionManager already exists");
   }
+  extensionManager = new ExtensionManager();
+  initContext = extensionManager.discoverExtensions(classes, 
systemBundle, bundles);
--- End diff --

ok


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133619468
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -121,33 +109,18 @@ private void unInit() {
* @throws IllegalStateException when already initialized with a given 
set of extension directories
* and extensionDirs does not match
*/
-  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  public static void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
   BundleProperties props)
   throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
 if (extensionsDirs == null || fileSystemManager == null) {
   throw new NullPointerException("cannot have empty arguments");
 }
-InitContext ic = initContext;
-if (ic == null) {
-  synchronized (this) {
-ic = initContext;
-if (ic == null) {
-  initContext = load(fileSystemManager, extensionsDirs, props);
-  ic = initContext;
-}
-  }
-} else {
-  boolean matching = CollectionUtils
-  .isEqualCollection(initContext.extensionDirs, extensionsDirs);
-  if (!matching) {
-throw new IllegalStateException(
-"Cannot reinitialize and extension directories cannot change");
-  }
-  matching = initContext.properties.match(props);
-  if (!matching) {
-throw new IllegalStateException(
-"Cannot re-initialize and properties cannot change");
+synchronized (BundleClassLoaders.class) {
+  if (bundleClassLoaders != null) {
+throw new IllegalStateException("BundleClassloader already 
exists");
   }
+  bundleClassLoaders = new BundleClassLoaders();
+  initContext = bundleClassLoaders.load(fileSystemManager, 
extensionsDirs, props);
--- End diff --

ok


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133530431
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -103,61 +104,52 @@ private ExtensionManager(){}
   /**
* @return The singleton instance of the ExtensionManager
*/
-  public static ExtensionManager getInstance() {
-ExtensionManager result = extensionManager;
-if (result == null) {
-  synchronized (ExtensionManager.class) {
-result = extensionManager;
-if (result == null) {
-  extensionManager = new ExtensionManager();
-  result = extensionManager;
-}
+  public static ExtensionManager getInstance() throws 
NotInitializedException {
+synchronized (ExtensionManager.class) {
+  if (extensionManager == null) {
+throw new NotInitializedException("ExtensionManager not 
initialized");
   }
+  return extensionManager;
 }
-return result;
   }
 
   /**
* Uninitializes the ExtensionManager.
* TESTING ONLY
*/
+  @VisibleForTesting
   public static void reset() {
 synchronized (ExtensionManager.class) {
-  getInstance().forgetExtensions();
+  initContext = null;
   extensionManager = null;
 }
   }
 
-  private void forgetExtensions() {
-  initContext = null;
-  }
-
-
   /**
* Loads all extension class types that can be found on the bootstrap 
classloader and by creating
* classloaders for all BUNDLES found within the classpath.
*
* @param bundles the bundles to scan through in search of extensions
*/
-  public void init(final List classes, final Bundle systemBundle, 
final Set bundles)
+  public static void init(final List classes, final Bundle 
systemBundle, final Set bundles)
   throws NotInitializedException {
 
-InitContext ic = initContext;
-if (ic == null) {
-  synchronized (this) {
-ic = initContext;
-if (ic == null) {
-  initContext = discoverExtensions(classes, systemBundle, bundles);
-  ic = initContext;
-}
+if (systemBundle == null) {
+  throw new IllegalArgumentException("systemBundle is required");
+}
+
+synchronized (ExtensionManager.class) {
+  if (extensionManager != null) {
+throw new IllegalStateException("ExtensionManager already exists");
   }
+  extensionManager = new ExtensionManager();
+  initContext = extensionManager.discoverExtensions(classes, 
systemBundle, bundles);
--- End diff --

And to protect unsynchronized getInstance():
```java
  ExtensionManager em = new ExtensionManager();
  InitContext ic = em.discoverExtensions(classes, systemBundle, 
bundles);
  initContext = ic;
  extensionManager = em;
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133533234
  
--- Diff: 
bundles-lib/src/test/java/org/apache/metron/bundles/BundleThreadContextClassLoaderTest.java
 ---
@@ -27,11 +27,21 @@
 import org.apache.metron.bundles.bundle.Bundle;
 import org.apache.metron.bundles.util.BundleProperties;
 import org.apache.metron.bundles.util.FileSystemManagerFactory;
+import org.junit.After;
 import org.junit.AfterClass;
 import org.junit.Test;
 
 public class BundleThreadContextClassLoaderTest {
+  @AfterClass
+  public static void after() {
+BundleClassLoaders.reset();
+  }
--- End diff --

Did you intend to do `ExtensionManager.reset();` here also?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133527688
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -121,33 +109,18 @@ private void unInit() {
* @throws IllegalStateException when already initialized with a given 
set of extension directories
* and extensionDirs does not match
*/
-  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  public static void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
   BundleProperties props)
   throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
 if (extensionsDirs == null || fileSystemManager == null) {
   throw new NullPointerException("cannot have empty arguments");
 }
-InitContext ic = initContext;
-if (ic == null) {
-  synchronized (this) {
-ic = initContext;
-if (ic == null) {
-  initContext = load(fileSystemManager, extensionsDirs, props);
-  ic = initContext;
-}
-  }
-} else {
-  boolean matching = CollectionUtils
-  .isEqualCollection(initContext.extensionDirs, extensionsDirs);
-  if (!matching) {
-throw new IllegalStateException(
-"Cannot reinitialize and extension directories cannot change");
-  }
-  matching = initContext.properties.match(props);
-  if (!matching) {
-throw new IllegalStateException(
-"Cannot re-initialize and properties cannot change");
+synchronized (BundleClassLoaders.class) {
+  if (bundleClassLoaders != null) {
+throw new IllegalStateException("BundleClassloader already 
exists");
   }
+  bundleClassLoaders = new BundleClassLoaders();
+  initContext = bundleClassLoaders.load(fileSystemManager, 
extensionsDirs, props);
--- End diff --

So that getInstance() can safely not synchronize, do this:
```java
  BundleClassLoaders b = new BundleClassLoaders();
  InitContext ic = b.load(fileSystemManager, extensionsDirs, props);
  initContext = ic;
  bundleClassLoaders = b;
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133532210
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -95,19 +91,11 @@ public static BundleClassLoaders getInstance() {
*/
   public static void reset() {
--- End diff --

Please add @ VisibleForTesting annotation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133083369
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleSystem.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+import java.lang.invoke.MethodHandles;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.FileSystemManagerFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * High level interface to the Bundle System.  While you may want to use 
the lower level classes it
+ * is not required, as BundleSystem provides the base required interface 
for initializing the system
+ * and instantiating classes
+ */
+public class BundleSystem {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  /**
+   * Builder for a BundleSystem. only {@link BundleProperties} are 
required. Beyond that, the
+   * BundleProperties, if they are the only parameter must have archive 
extension and bundle
+   * extension types properties present.
+   */
+  public static class Builder {
+
+private BundleProperties properties;
+private FileSystemManager fileSystemManager;
+private List extensionClasses = new LinkedList<>();
+private Bundle systemBundle;
+
+/**
+ * The BundleProperties to use.  Unless other builder parameters 
override options
+ * (withExtensionClasses ), they must have archive extension and 
bundle extensions types
+ * specified
+ *
+ * @param properties The BundleProperties
+ * @return Builder
+ */
+public Builder withBundleProperties(BundleProperties properties) {
+  this.properties = properties;
+  return this;
+}
+
+/**
+ * Provide a {@link FileSystemManager} to overide the default
+ *
+ * @param fileSystemManager override
+ * @return Builder
+ */
+public Builder withFileSystemManager(FileSystemManager 
fileSystemManager) {
+  this.fileSystemManager = fileSystemManager;
+  return this;
+}
+
+/**
+ * Provide Extension Classes.  If not provided with this override then 
the classes will be
+ * configured from the BundleProperties. If provided, the properties 
file will not be used.
+ *
+ * @param extensionClasses override
+ * @return Builder
+ */
+public Builder withExtensionClasses(List extensionClasses) {
+  this.extensionClasses.addAll(extensionClasses);
+  return this;
+}
+
+/**
+ * Provide a SystemBundle.  If not provided with this override then 
the default SystemBundle
+ * will be created.
+ */
+public Builder withSystemBundle(Bundle systemBundle) {
+  this.systemBundle = systemBundle;
+  return this;
+}
+
+/**
+ * Builds a new BundleSystem.
+ *
+ * @return BundleSystem
+ * @throws NotInitializedException if any errors happen during build
+ */
+public BundleSystem build() throws NotInitializedException {
+  if (this.properties == null) {
+throw new IllegalArgumentException("BundleProperties are 
required");
--- End diff --

I will make it more clear they will not be used for the classes, but will 
be used for other things


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133081975
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleSystem.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+import java.lang.invoke.MethodHandles;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.FileSystemManagerFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * High level interface to the Bundle System.  While you may want to use 
the lower level classes it
+ * is not required, as BundleSystem provides the base required interface 
for initializing the system
+ * and instantiating classes
+ */
+public class BundleSystem {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  /**
+   * Builder for a BundleSystem. only {@link BundleProperties} are 
required. Beyond that, the
+   * BundleProperties, if they are the only parameter must have archive 
extension and bundle
+   * extension types properties present.
+   */
+  public static class Builder {
+
+private BundleProperties properties;
+private FileSystemManager fileSystemManager;
+private List extensionClasses = new LinkedList<>();
+private Bundle systemBundle;
+
+/**
+ * The BundleProperties to use.  Unless other builder parameters 
override options
+ * (withExtensionClasses ), they must have archive extension and 
bundle extensions types
+ * specified
+ *
+ * @param properties The BundleProperties
+ * @return Builder
+ */
+public Builder withBundleProperties(BundleProperties properties) {
+  this.properties = properties;
+  return this;
+}
+
+/**
+ * Provide a {@link FileSystemManager} to overide the default
+ *
+ * @param fileSystemManager override
+ * @return Builder
+ */
+public Builder withFileSystemManager(FileSystemManager 
fileSystemManager) {
+  this.fileSystemManager = fileSystemManager;
+  return this;
+}
+
+/**
+ * Provide Extension Classes.  If not provided with this override then 
the classes will be
+ * configured from the BundleProperties. If provided, the properties 
file will not be used.
+ *
+ * @param extensionClasses override
+ * @return Builder
+ */
+public Builder withExtensionClasses(List extensionClasses) {
+  this.extensionClasses.addAll(extensionClasses);
+  return this;
+}
+
+/**
+ * Provide a SystemBundle.  If not provided with this override then 
the default SystemBundle
+ * will be created.
+ */
+public Builder withSystemBundle(Bundle systemBundle) {
+  this.systemBundle = systemBundle;
+  return this;
+}
+
+/**
+ * Builds a new BundleSystem.
+ *
+ * @return BundleSystem
+ * @throws NotInitializedException if any errors happen during build
+ */
+public BundleSystem build() throws NotInitializedException {
+  if (this.properties == null) {
+throw new IllegalArgumentException("BundleProperties are 
required");
--- End diff --

Ah, then please correct the above comment on withExtensionClasses() that 
says "If \[ExtensionClasses\] provided, the properties file will not be used.".


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133081623
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/util/ImmutableCollectionUtils.java
 ---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles.util;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Utility class for creating immutable versions of collections, where the 
inner collection needs to
+ * be immutable as well.
+ *
+ * {@see com.google.common.collect}
+ */
+public class ImmutableCollectionUtils {
--- End diff --

accepted.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133079554
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all 
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static volatile ExtensionManager extensionManager;
+  private volatile InitContext initContext;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that 
implement the interface
+private final Map definitionMap;
+private final Map classNameBundleLookup;
+private final Map 
bundleCoordinateBundleLookup;
+private final Map classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map instanceClassloaderLookup;
+
+private InitContext(Map definitionMap,
+Map classNameBundleLookup,
+Map bundleCoordinateBundleLookup,
+Map classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map instanceClassloaderLookup) {
+
+  this.definitionMap = 
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+  this.classNameBundleLookup = ImmutableCollectionUtils
+  .immutableMapOfLists(classNameBundleLookup);
+  this.bundleCoordinateBundleLookup = 
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+  this.classLoaderBundleLookup = 
ImmutableMap.copyOf(classLoaderBundleLookup);
+  this.requiresInstanceClassLoading = 
ImmutableSet.copyOf(requiresInstanceClassLoading);
+  this.instanceClassloaderLookup = new 
ConcurrentHashMap<>(instanceClassloaderLookup);
+}
+  }
+
+  private 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133079524
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all 
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static volatile ExtensionManager extensionManager;
+  private volatile InitContext initContext;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that 
implement the interface
+private final Map definitionMap;
+private final Map classNameBundleLookup;
+private final Map 
bundleCoordinateBundleLookup;
+private final Map classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map instanceClassloaderLookup;
+
+private InitContext(Map definitionMap,
+Map classNameBundleLookup,
+Map bundleCoordinateBundleLookup,
+Map classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map instanceClassloaderLookup) {
+
+  this.definitionMap = 
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+  this.classNameBundleLookup = ImmutableCollectionUtils
+  .immutableMapOfLists(classNameBundleLookup);
+  this.bundleCoordinateBundleLookup = 
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+  this.classLoaderBundleLookup = 
ImmutableMap.copyOf(classLoaderBundleLookup);
+  this.requiresInstanceClassLoading = 
ImmutableSet.copyOf(requiresInstanceClassLoading);
+  this.instanceClassloaderLookup = new 
ConcurrentHashMap<>(instanceClassloaderLookup);
+}
+  }
+
+  private 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133079476
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all 
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static volatile ExtensionManager extensionManager;
+  private volatile InitContext initContext;
--- End diff --

done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133079584
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java ---
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+
+import java.util.*;
+import java.util.function.BiFunction;
+
+/**
+ * The ExtensionMapping represents a mapping of the extensions available 
to the system.
+ * It is the product of the BundleMapper.
+ *
+ * It is NOT used at runtime for loading extensions, rather it may be used 
by a system to
--- End diff --

done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133079513
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all 
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static volatile ExtensionManager extensionManager;
+  private volatile InitContext initContext;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that 
implement the interface
+private final Map definitionMap;
+private final Map classNameBundleLookup;
+private final Map 
bundleCoordinateBundleLookup;
+private final Map classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map instanceClassloaderLookup;
+
+private InitContext(Map definitionMap,
+Map classNameBundleLookup,
+Map bundleCoordinateBundleLookup,
+Map classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map instanceClassloaderLookup) {
+
+  this.definitionMap = 
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+  this.classNameBundleLookup = ImmutableCollectionUtils
+  .immutableMapOfLists(classNameBundleLookup);
+  this.bundleCoordinateBundleLookup = 
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+  this.classLoaderBundleLookup = 
ImmutableMap.copyOf(classLoaderBundleLookup);
+  this.requiresInstanceClassLoading = 
ImmutableSet.copyOf(requiresInstanceClassLoading);
+  this.instanceClassloaderLookup = new 
ConcurrentHashMap<>(instanceClassloaderLookup);
+}
+  }
+
+  private 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133079462
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all 
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
--- End diff --

done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133079502
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all 
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static volatile ExtensionManager extensionManager;
+  private volatile InitContext initContext;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that 
implement the interface
+private final Map definitionMap;
+private final Map classNameBundleLookup;
+private final Map 
bundleCoordinateBundleLookup;
+private final Map classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map instanceClassloaderLookup;
+
+private InitContext(Map definitionMap,
+Map classNameBundleLookup,
+Map bundleCoordinateBundleLookup,
+Map classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map instanceClassloaderLookup) {
+
+  this.definitionMap = 
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+  this.classNameBundleLookup = ImmutableCollectionUtils
+  .immutableMapOfLists(classNameBundleLookup);
+  this.bundleCoordinateBundleLookup = 
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+  this.classLoaderBundleLookup = 
ImmutableMap.copyOf(classLoaderBundleLookup);
+  this.requiresInstanceClassLoading = 
ImmutableSet.copyOf(requiresInstanceClassLoading);
+  this.instanceClassloaderLookup = new 
ConcurrentHashMap<>(instanceClassloaderLookup);
+}
+  }
+
+  private 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133079542
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all 
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static volatile ExtensionManager extensionManager;
+  private volatile InitContext initContext;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that 
implement the interface
+private final Map definitionMap;
+private final Map classNameBundleLookup;
+private final Map 
bundleCoordinateBundleLookup;
+private final Map classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map instanceClassloaderLookup;
+
+private InitContext(Map definitionMap,
+Map classNameBundleLookup,
+Map bundleCoordinateBundleLookup,
+Map classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map instanceClassloaderLookup) {
+
+  this.definitionMap = 
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+  this.classNameBundleLookup = ImmutableCollectionUtils
+  .immutableMapOfLists(classNameBundleLookup);
+  this.bundleCoordinateBundleLookup = 
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+  this.classLoaderBundleLookup = 
ImmutableMap.copyOf(classLoaderBundleLookup);
+  this.requiresInstanceClassLoading = 
ImmutableSet.copyOf(requiresInstanceClassLoading);
+  this.instanceClassloaderLookup = new 
ConcurrentHashMap<>(instanceClassloaderLookup);
+}
+  }
+
+  private 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133079324
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,376 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  /**
+   * Holds the context from {@code BundleClassLoaders} initialization,
+   * being the coordinate to bundle mapping.
+   *
+   * After initialization these are not changed, and as such they
+   * are immutable.
+   *
+   */
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+private final BundleProperties properties;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles,
+final BundleProperties properties) {
+  this.extensionDirs = ImmutableList.copyOf(extensionDirs);
+  this.bundles = ImmutableMap.copyOf(bundles);
+  this.properties = properties;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = new BundleClassLoaders();
+  result = bundleClassLoaders;
+}
+  }
+}
+return result;
+  }
--- End diff --

done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133079379
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * THREAD SAFE
--- End diff --

gone


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133079362
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,376 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  /**
+   * Holds the context from {@code BundleClassLoaders} initialization,
+   * being the coordinate to bundle mapping.
+   *
+   * After initialization these are not changed, and as such they
+   * are immutable.
+   *
+   */
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+private final BundleProperties properties;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles,
+final BundleProperties properties) {
+  this.extensionDirs = ImmutableList.copyOf(extensionDirs);
+  this.bundles = ImmutableMap.copyOf(bundles);
+  this.properties = properties;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = new BundleClassLoaders();
+  result = bundleClassLoaders;
+}
+  }
+}
+return result;
+  }
+
+  /**
+   * Uninitializes the BundleClassloaders.  After calling this 
init must be called
+   * before the rest of the methods are called afterwards.
+   * This is for TESTING ONLY at this time.  Reset does not unload or 
clear any loaded classloaders.
+   */
+  public static void reset() {
+synchronized (BundleClassLoaders.class) {
+  getInstance().unInit();
+  bundleClassLoaders = null;
+}
+  }
+
+  private void unInit() {
+synchronized (this) {
+  if(initContext != null) {
+initContext = null;
+  }
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called.  Multiple calls to 
this method will have no effect,
+   * unless a different set of extension directories is passed, which will 
result in an IllegaStateException
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133078904
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleSystem.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+import java.lang.invoke.MethodHandles;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.FileSystemManagerFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * High level interface to the Bundle System.  While you may want to use 
the lower level classes it
+ * is not required, as BundleSystem provides the base required interface 
for initializing the system
+ * and instantiating classes
+ */
+public class BundleSystem {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  /**
+   * Builder for a BundleSystem. only {@link BundleProperties} are 
required. Beyond that, the
+   * BundleProperties, if they are the only parameter must have archive 
extension and bundle
+   * extension types properties present.
+   */
+  public static class Builder {
+
+private BundleProperties properties;
+private FileSystemManager fileSystemManager;
+private List extensionClasses = new LinkedList<>();
+private Bundle systemBundle;
+
+/**
+ * The BundleProperties to use.  Unless other builder parameters 
override options
+ * (withExtensionClasses ), they must have archive extension and 
bundle extensions types
+ * specified
+ *
+ * @param properties The BundleProperties
+ * @return Builder
+ */
+public Builder withBundleProperties(BundleProperties properties) {
+  this.properties = properties;
+  return this;
+}
+
+/**
+ * Provide a {@link FileSystemManager} to overide the default
+ *
+ * @param fileSystemManager override
+ * @return Builder
+ */
+public Builder withFileSystemManager(FileSystemManager 
fileSystemManager) {
+  this.fileSystemManager = fileSystemManager;
+  return this;
+}
+
+/**
+ * Provide Extension Classes.  If not provided with this override then 
the classes will be
+ * configured from the BundleProperties. If provided, the properties 
file will not be used.
+ *
+ * @param extensionClasses override
+ * @return Builder
+ */
+public Builder withExtensionClasses(List extensionClasses) {
+  this.extensionClasses.addAll(extensionClasses);
+  return this;
+}
+
+/**
+ * Provide a SystemBundle.  If not provided with this override then 
the default SystemBundle
+ * will be created.
+ */
+public Builder withSystemBundle(Bundle systemBundle) {
+  this.systemBundle = systemBundle;
+  return this;
+}
+
+/**
+ * Builds a new BundleSystem.
+ *
+ * @return BundleSystem
+ * @throws NotInitializedException if any errors happen during build
+ */
+public BundleSystem build() throws NotInitializedException {
+  if (this.properties == null) {
+throw new IllegalArgumentException("BundleProperties are 
required");
--- End diff --

We need the properties for other things


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133078228
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all 
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static volatile ExtensionManager extensionManager;
+  private volatile InitContext initContext;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that 
implement the interface
+private final Map definitionMap;
+private final Map classNameBundleLookup;
+private final Map 
bundleCoordinateBundleLookup;
+private final Map classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map instanceClassloaderLookup;
+
+private InitContext(Map definitionMap,
+Map classNameBundleLookup,
+Map bundleCoordinateBundleLookup,
+Map classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map instanceClassloaderLookup) {
+
+  this.definitionMap = 
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+  this.classNameBundleLookup = ImmutableCollectionUtils
+  .immutableMapOfLists(classNameBundleLookup);
+  this.bundleCoordinateBundleLookup = 
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+  this.classLoaderBundleLookup = 
ImmutableMap.copyOf(classLoaderBundleLookup);
+  this.requiresInstanceClassLoading = 
ImmutableSet.copyOf(requiresInstanceClassLoading);
+  this.instanceClassloaderLookup = new 
ConcurrentHashMap<>(instanceClassloaderLookup);
+}
--- End diff --

Yes, 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133077758
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * THREAD SAFE
+ */
+public class BundleThreadContextClassLoader extends URLClassLoader {
+static final Logger LOG = 
LoggerFactory.getLogger(BundleThreadContextClassLoader.class);
+static final ContextSecurityManager contextSecurityManager = new 
ContextSecurityManager();
+private final ClassLoader forward = ClassLoader.getSystemClassLoader();
+
+private BundleThreadContextClassLoader() {
+super(new URL[0]);
+}
+
+@Override
+public void clearAssertionStatus() {
+lookupClassLoader().clearAssertionStatus();
+}
+
+@Override
+public URL getResource(String name) {
+return lookupClassLoader().getResource(name);
+}
+
+@Override
+public InputStream getResourceAsStream(String name) {
+return lookupClassLoader().getResourceAsStream(name);
+}
+
+@Override
+public Enumeration getResources(String name) throws IOException {
+return lookupClassLoader().getResources(name);
+}
+
+@Override
+public Class loadClass(String name) throws ClassNotFoundException {
+return lookupClassLoader().loadClass(name);
+}
+
+@Override
+public void setClassAssertionStatus(String className, boolean enabled) 
{
+lookupClassLoader().setClassAssertionStatus(className, enabled);
+}
+
+@Override
+public void setDefaultAssertionStatus(boolean enabled) {
+lookupClassLoader().setDefaultAssertionStatus(enabled);
+}
+
+@Override
+public void setPackageAssertionStatus(String packageName, boolean 
enabled) {
+lookupClassLoader().setPackageAssertionStatus(packageName, 
enabled);
+}
+
+private ClassLoader lookupClassLoader() {
+final Class[] classStack = 
contextSecurityManager.getExecutionStack();
+
+for (Class currentClass : classStack) {
+final Class bundleClass = findBundleClass(currentClass);
+if (bundleClass != null) {
+final ClassLoader desiredClassLoader = 
bundleClass.getClassLoader();
+
+// When new Threads are created, the new Thread inherits 
the ClassLoaderContext of
+// the caller. However, the call stack of that new Thread 
may not trace back to any app-specific
+// code. Therefore, the BundleThreadContextClassLoader 
will be unable to find the appropriate Bundle
+// ClassLoader. As a result, we want to set the 
ContextClassLoader to the Bundle ClassLoader that
+// contains the class or resource that we are looking for.
+// This locks the current Thread into the appropriate 
Bundle ClassLoader Context. The framework will change
+// the ContextClassLoader back to the 
BundleThreadContextClassLoader as appropriate via the
+//
+// TL;DR
+// We need to make sure the classloader for the thread is 
setup correctly to use the 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133076687
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * THREAD SAFE
+ */
+public class BundleThreadContextClassLoader extends URLClassLoader {
+static final Logger LOG = 
LoggerFactory.getLogger(BundleThreadContextClassLoader.class);
+static final ContextSecurityManager contextSecurityManager = new 
ContextSecurityManager();
+private final ClassLoader forward = ClassLoader.getSystemClassLoader();
+
--- End diff --

If we can't get rid of all the static init, at least we can get rid of as 
much as possible.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133051603
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all 
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static volatile ExtensionManager extensionManager;
+  private volatile InitContext initContext;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that 
implement the interface
+private final Map definitionMap;
+private final Map classNameBundleLookup;
+private final Map 
bundleCoordinateBundleLookup;
+private final Map classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map instanceClassloaderLookup;
+
+private InitContext(Map definitionMap,
+Map classNameBundleLookup,
+Map bundleCoordinateBundleLookup,
+Map classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map instanceClassloaderLookup) {
+
+  this.definitionMap = 
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+  this.classNameBundleLookup = ImmutableCollectionUtils
+  .immutableMapOfLists(classNameBundleLookup);
+  this.bundleCoordinateBundleLookup = 
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+  this.classLoaderBundleLookup = 
ImmutableMap.copyOf(classLoaderBundleLookup);
+  this.requiresInstanceClassLoading = 
ImmutableSet.copyOf(requiresInstanceClassLoading);
+  this.instanceClassloaderLookup = new 
ConcurrentHashMap<>(instanceClassloaderLookup);
+}
+  }
+
+  private 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133033964
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,376 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
--- End diff --

Since BundleClassLoaders is a singleton, this (initContext) should be 
static too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133049444
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all 
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static volatile ExtensionManager extensionManager;
+  private volatile InitContext initContext;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that 
implement the interface
+private final Map definitionMap;
+private final Map classNameBundleLookup;
+private final Map 
bundleCoordinateBundleLookup;
+private final Map classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map instanceClassloaderLookup;
+
+private InitContext(Map definitionMap,
+Map classNameBundleLookup,
+Map bundleCoordinateBundleLookup,
+Map classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map instanceClassloaderLookup) {
+
+  this.definitionMap = 
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+  this.classNameBundleLookup = ImmutableCollectionUtils
+  .immutableMapOfLists(classNameBundleLookup);
+  this.bundleCoordinateBundleLookup = 
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+  this.classLoaderBundleLookup = 
ImmutableMap.copyOf(classLoaderBundleLookup);
+  this.requiresInstanceClassLoading = 
ImmutableSet.copyOf(requiresInstanceClassLoading);
+  this.instanceClassloaderLookup = new 
ConcurrentHashMap<>(instanceClassloaderLookup);
+}
--- End diff --

Good 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133037865
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,376 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  /**
+   * Holds the context from {@code BundleClassLoaders} initialization,
+   * being the coordinate to bundle mapping.
+   *
+   * After initialization these are not changed, and as such they
+   * are immutable.
+   *
+   */
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+private final BundleProperties properties;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles,
+final BundleProperties properties) {
+  this.extensionDirs = ImmutableList.copyOf(extensionDirs);
+  this.bundles = ImmutableMap.copyOf(bundles);
+  this.properties = properties;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = new BundleClassLoaders();
+  result = bundleClassLoaders;
+}
+  }
+}
+return result;
+  }
+
+  /**
+   * Uninitializes the BundleClassloaders.  After calling this 
init must be called
+   * before the rest of the methods are called afterwards.
+   * This is for TESTING ONLY at this time.  Reset does not unload or 
clear any loaded classloaders.
+   */
+  public static void reset() {
+synchronized (BundleClassLoaders.class) {
+  getInstance().unInit();
+  bundleClassLoaders = null;
+}
+  }
+
+  private void unInit() {
+synchronized (this) {
+  if(initContext != null) {
+initContext = null;
+  }
--- End diff --

Recommend combining reset and unInit methods.  This avoids the issues of
- having reset static while unInit is per-instance (doesn't matter which 
since is singleton, but confusing to use both)
- using double synchronize calls (ok even with both static or both 
per-instance, because Java locks are reentrant, but unnecessary and raises 
concerns in paranoid reviewers like me)


---
If your project is set up for it, you can reply to this email and have your
reply appear on 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133043905
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * THREAD SAFE
+ */
+public class BundleThreadContextClassLoader extends URLClassLoader {
+static final Logger LOG = 
LoggerFactory.getLogger(BundleThreadContextClassLoader.class);
+static final ContextSecurityManager contextSecurityManager = new 
ContextSecurityManager();
+private final ClassLoader forward = ClassLoader.getSystemClassLoader();
+
--- End diff --

I very much like that you made BundleThreadContextClassLoader statically 
self-initializing, by using 
ExtensionManager.getInstance().getExtensionClasses() instead of keeping a copy 
here that had to be initialized.  Good job.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133046812
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * THREAD SAFE
+ */
+public class BundleThreadContextClassLoader extends URLClassLoader {
+static final Logger LOG = 
LoggerFactory.getLogger(BundleThreadContextClassLoader.class);
+static final ContextSecurityManager contextSecurityManager = new 
ContextSecurityManager();
+private final ClassLoader forward = ClassLoader.getSystemClassLoader();
+
+private BundleThreadContextClassLoader() {
+super(new URL[0]);
+}
+
+@Override
+public void clearAssertionStatus() {
+lookupClassLoader().clearAssertionStatus();
+}
+
+@Override
+public URL getResource(String name) {
+return lookupClassLoader().getResource(name);
+}
+
+@Override
+public InputStream getResourceAsStream(String name) {
+return lookupClassLoader().getResourceAsStream(name);
+}
+
+@Override
+public Enumeration getResources(String name) throws IOException {
+return lookupClassLoader().getResources(name);
+}
+
+@Override
+public Class loadClass(String name) throws ClassNotFoundException {
+return lookupClassLoader().loadClass(name);
+}
+
+@Override
+public void setClassAssertionStatus(String className, boolean enabled) 
{
+lookupClassLoader().setClassAssertionStatus(className, enabled);
+}
+
+@Override
+public void setDefaultAssertionStatus(boolean enabled) {
+lookupClassLoader().setDefaultAssertionStatus(enabled);
+}
+
+@Override
+public void setPackageAssertionStatus(String packageName, boolean 
enabled) {
+lookupClassLoader().setPackageAssertionStatus(packageName, 
enabled);
+}
+
+private ClassLoader lookupClassLoader() {
+final Class[] classStack = 
contextSecurityManager.getExecutionStack();
+
+for (Class currentClass : classStack) {
+final Class bundleClass = findBundleClass(currentClass);
+if (bundleClass != null) {
+final ClassLoader desiredClassLoader = 
bundleClass.getClassLoader();
+
+// When new Threads are created, the new Thread inherits 
the ClassLoaderContext of
+// the caller. However, the call stack of that new Thread 
may not trace back to any app-specific
+// code. Therefore, the BundleThreadContextClassLoader 
will be unable to find the appropriate Bundle
+// ClassLoader. As a result, we want to set the 
ContextClassLoader to the Bundle ClassLoader that
+// contains the class or resource that we are looking for.
+// This locks the current Thread into the appropriate 
Bundle ClassLoader Context. The framework will change
+// the ContextClassLoader back to the 
BundleThreadContextClassLoader as appropriate via the
+//
+// TL;DR
+// We need to make sure the classloader for the thread is 
setup correctly to use the 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133052945
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all 
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static volatile ExtensionManager extensionManager;
+  private volatile InitContext initContext;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that 
implement the interface
+private final Map definitionMap;
+private final Map classNameBundleLookup;
+private final Map 
bundleCoordinateBundleLookup;
+private final Map classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map instanceClassloaderLookup;
+
+private InitContext(Map definitionMap,
+Map classNameBundleLookup,
+Map bundleCoordinateBundleLookup,
+Map classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map instanceClassloaderLookup) {
+
+  this.definitionMap = 
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+  this.classNameBundleLookup = ImmutableCollectionUtils
+  .immutableMapOfLists(classNameBundleLookup);
+  this.bundleCoordinateBundleLookup = 
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+  this.classLoaderBundleLookup = 
ImmutableMap.copyOf(classLoaderBundleLookup);
+  this.requiresInstanceClassLoading = 
ImmutableSet.copyOf(requiresInstanceClassLoading);
+  this.instanceClassloaderLookup = new 
ConcurrentHashMap<>(instanceClassloaderLookup);
+}
+  }
+
+  private 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133049087
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all 
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static volatile ExtensionManager extensionManager;
+  private volatile InitContext initContext;
--- End diff --

make initContext static, see below


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133062854
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java ---
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+
+import java.util.*;
+import java.util.function.BiFunction;
+
+/**
+ * The ExtensionMapping represents a mapping of the extensions available 
to the system.
+ * It is the product of the BundleMapper.
+ *
+ * It is NOT used at runtime for loading extensions, rather it may be used 
by a system to
--- End diff --

Please note which method does load extensions at startup.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133049372
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all 
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static volatile ExtensionManager extensionManager;
+  private volatile InitContext initContext;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that 
implement the interface
+private final Map definitionMap;
+private final Map classNameBundleLookup;
+private final Map 
bundleCoordinateBundleLookup;
+private final Map classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map instanceClassloaderLookup;
+
+private InitContext(Map definitionMap,
+Map classNameBundleLookup,
+Map bundleCoordinateBundleLookup,
+Map classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map instanceClassloaderLookup) {
+
+  this.definitionMap = 
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+  this.classNameBundleLookup = ImmutableCollectionUtils
+  .immutableMapOfLists(classNameBundleLookup);
+  this.bundleCoordinateBundleLookup = 
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+  this.classLoaderBundleLookup = 
ImmutableMap.copyOf(classLoaderBundleLookup);
+  this.requiresInstanceClassLoading = 
ImmutableSet.copyOf(requiresInstanceClassLoading);
+  this.instanceClassloaderLookup = new 
ConcurrentHashMap<>(instanceClassloaderLookup);
+}
+  }
+
+  private 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133060635
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleSystem.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+import java.lang.invoke.MethodHandles;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.FileSystemManagerFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * High level interface to the Bundle System.  While you may want to use 
the lower level classes it
+ * is not required, as BundleSystem provides the base required interface 
for initializing the system
+ * and instantiating classes
+ */
+public class BundleSystem {
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+  /**
+   * Builder for a BundleSystem. only {@link BundleProperties} are 
required. Beyond that, the
+   * BundleProperties, if they are the only parameter must have archive 
extension and bundle
+   * extension types properties present.
+   */
+  public static class Builder {
+
+private BundleProperties properties;
+private FileSystemManager fileSystemManager;
+private List extensionClasses = new LinkedList<>();
+private Bundle systemBundle;
+
+/**
+ * The BundleProperties to use.  Unless other builder parameters 
override options
+ * (withExtensionClasses ), they must have archive extension and 
bundle extensions types
+ * specified
+ *
+ * @param properties The BundleProperties
+ * @return Builder
+ */
+public Builder withBundleProperties(BundleProperties properties) {
+  this.properties = properties;
+  return this;
+}
+
+/**
+ * Provide a {@link FileSystemManager} to overide the default
+ *
+ * @param fileSystemManager override
+ * @return Builder
+ */
+public Builder withFileSystemManager(FileSystemManager 
fileSystemManager) {
+  this.fileSystemManager = fileSystemManager;
+  return this;
+}
+
+/**
+ * Provide Extension Classes.  If not provided with this override then 
the classes will be
+ * configured from the BundleProperties. If provided, the properties 
file will not be used.
+ *
+ * @param extensionClasses override
+ * @return Builder
+ */
+public Builder withExtensionClasses(List extensionClasses) {
+  this.extensionClasses.addAll(extensionClasses);
+  return this;
+}
+
+/**
+ * Provide a SystemBundle.  If not provided with this override then 
the default SystemBundle
+ * will be created.
+ */
+public Builder withSystemBundle(Bundle systemBundle) {
+  this.systemBundle = systemBundle;
+  return this;
+}
+
+/**
+ * Builds a new BundleSystem.
+ *
+ * @return BundleSystem
+ * @throws NotInitializedException if any errors happen during build
+ */
+public BundleSystem build() throws NotInitializedException {
+  if (this.properties == null) {
+throw new IllegalArgumentException("BundleProperties are 
required");
--- End diff --

But you also said the only use of properties file is for the extension 
classes.  This suggests an either/or with the "extensionClasses" field.  But if 
you feel making properties mandatory is future-proofing, that's okay.


---
If your project is set up for it, you can reply to this email and have your

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133035166
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,376 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  /**
+   * Holds the context from {@code BundleClassLoaders} initialization,
+   * being the coordinate to bundle mapping.
+   *
+   * After initialization these are not changed, and as such they
+   * are immutable.
+   *
+   */
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+private final BundleProperties properties;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles,
+final BundleProperties properties) {
+  this.extensionDirs = ImmutableList.copyOf(extensionDirs);
+  this.bundles = ImmutableMap.copyOf(bundles);
+  this.properties = properties;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = new BundleClassLoaders();
+  result = bundleClassLoaders;
+}
+  }
+}
+return result;
+  }
+
+  /**
+   * Uninitializes the BundleClassloaders.  After calling this 
init must be called
+   * before the rest of the methods are called afterwards.
+   * This is for TESTING ONLY at this time.  Reset does not unload or 
clear any loaded classloaders.
+   */
+  public static void reset() {
+synchronized (BundleClassLoaders.class) {
+  getInstance().unInit();
+  bundleClassLoaders = null;
+}
+  }
+
+  private void unInit() {
+synchronized (this) {
+  if(initContext != null) {
+initContext = null;
+  }
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called.  Multiple calls to 
this method will have no effect,
+   * unless a different set of extension directories is passed, which will 
result in an IllegaStateException
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133048319
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * THREAD SAFE
--- End diff --

Is "THREAD SAFE" a declaration or a design recommendation?  In either case, 
it begs supporting information.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133050317
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all 
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static volatile ExtensionManager extensionManager;
+  private volatile InitContext initContext;
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that 
implement the interface
+private final Map definitionMap;
+private final Map classNameBundleLookup;
+private final Map 
bundleCoordinateBundleLookup;
+private final Map classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map instanceClassloaderLookup;
+
+private InitContext(Map definitionMap,
+Map classNameBundleLookup,
+Map bundleCoordinateBundleLookup,
+Map classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map instanceClassloaderLookup) {
+
+  this.definitionMap = 
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+  this.classNameBundleLookup = ImmutableCollectionUtils
+  .immutableMapOfLists(classNameBundleLookup);
+  this.bundleCoordinateBundleLookup = 
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+  this.classLoaderBundleLookup = 
ImmutableMap.copyOf(classLoaderBundleLookup);
+  this.requiresInstanceClassLoading = 
ImmutableSet.copyOf(requiresInstanceClassLoading);
+  this.instanceClassloaderLookup = new 
ConcurrentHashMap<>(instanceClassloaderLookup);
+}
+  }
+
+  private 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133047499
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java
 ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * THREAD SAFE
+ */
+public class BundleThreadContextClassLoader extends URLClassLoader {
+static final Logger LOG = 
LoggerFactory.getLogger(BundleThreadContextClassLoader.class);
+static final ContextSecurityManager contextSecurityManager = new 
ContextSecurityManager();
+private final ClassLoader forward = ClassLoader.getSystemClassLoader();
+
+private BundleThreadContextClassLoader() {
+super(new URL[0]);
+}
+
+@Override
+public void clearAssertionStatus() {
+lookupClassLoader().clearAssertionStatus();
+}
+
+@Override
+public URL getResource(String name) {
+return lookupClassLoader().getResource(name);
+}
+
+@Override
+public InputStream getResourceAsStream(String name) {
+return lookupClassLoader().getResourceAsStream(name);
+}
+
+@Override
+public Enumeration getResources(String name) throws IOException {
+return lookupClassLoader().getResources(name);
+}
+
+@Override
+public Class loadClass(String name) throws ClassNotFoundException {
+return lookupClassLoader().loadClass(name);
+}
+
+@Override
+public void setClassAssertionStatus(String className, boolean enabled) 
{
+lookupClassLoader().setClassAssertionStatus(className, enabled);
+}
+
+@Override
+public void setDefaultAssertionStatus(boolean enabled) {
+lookupClassLoader().setDefaultAssertionStatus(enabled);
+}
+
+@Override
+public void setPackageAssertionStatus(String packageName, boolean 
enabled) {
+lookupClassLoader().setPackageAssertionStatus(packageName, 
enabled);
+}
+
+private ClassLoader lookupClassLoader() {
+final Class[] classStack = 
contextSecurityManager.getExecutionStack();
+
+for (Class currentClass : classStack) {
+final Class bundleClass = findBundleClass(currentClass);
+if (bundleClass != null) {
+final ClassLoader desiredClassLoader = 
bundleClass.getClassLoader();
+
+// When new Threads are created, the new Thread inherits 
the ClassLoaderContext of
+// the caller. However, the call stack of that new Thread 
may not trace back to any app-specific
+// code. Therefore, the BundleThreadContextClassLoader 
will be unable to find the appropriate Bundle
+// ClassLoader. As a result, we want to set the 
ContextClassLoader to the Bundle ClassLoader that
+// contains the class or resource that we are looking for.
+// This locks the current Thread into the appropriate 
Bundle ClassLoader Context. The framework will change
+// the ContextClassLoader back to the 
BundleThreadContextClassLoader as appropriate via the
+//
+// TL;DR
+// We need to make sure the classloader for the thread is 
setup correctly to use the 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133050916
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+
+import java.util.*;
+import java.util.function.BiFunction;
+
+public class ExtensionMapping {
+
+  private final Map> 
extensionNameMap = new HashMap<>();
--- End diff --

Which I have now done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133061877
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,436 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * Scans through the classpath to load all extension components using the 
service provider API and
+ * running through all classloaders (root, BUNDLEs).
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  // Maps a service definition (interface) to those classes that implement 
the interface
+  private static final Map definitionMap = new 
HashMap<>();
+
+  private static final Map classNameBundleLookup = 
new HashMap<>();
+  private static final Map 
bundleCoordinateBundleLookup = new HashMap<>();
+  private static final Map classLoaderBundleLookup = 
new HashMap<>();
+
+  private static final Set requiresInstanceClassLoading = new 
HashSet<>();
+  private static final Map instanceClassloaderLookup 
= new ConcurrentHashMap<>();
+
+  private static AtomicBoolean inited = new AtomicBoolean(false);
+
+  // should initialize class definitions
+  public static void initClassDefinitions(final List classes) {
+if (classes != null) {
+  for (Class clazz : classes) {
+definitionMap.put(clazz, new HashSet<>());
+  }
+}
+inited.set(true);
+  }
+
+  public static void resetClassDefinitions() {
+definitionMap.clear();
+inited.set(false);
+  }
+
+  /**
+   * Loads all extension class types that can be found on the bootstrap 
classloader and by creating
+   * classloaders for all BUNDLES found within the classpath.
+   *
+   * @param bundles the bundles to scan through in search of extensions
+   */
+  public static void discoverExtensions(final Bundle systemBundle, final 
Set bundles)
+  throws NotInitializedException {
+checkInitialized();
+// get the current context class loader
+ClassLoader currentContextClassLoader = 
Thread.currentThread().getContextClassLoader();
+
+// load the system bundle first so that any extensions found in JARs 
directly in lib will be registered as
+// being from the system bundle and not from all the other Bundles
+loadExtensions(systemBundle);
+
bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(),
 systemBundle);
+
+// 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r133033031
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+
+import java.util.*;
+import java.util.function.BiFunction;
+
+public class ExtensionMapping {
+
+  private final Map> 
extensionNameMap = new HashMap<>();
--- End diff --

Note - that with the BundleSystem, the properties file IS now used for 
loading, unlike before - IF the classes are not explicitly set.

I will have to write something to explain the ExtensionMappings vs. the 
ExtensionManager


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132820993
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,436 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * Scans through the classpath to load all extension components using the 
service provider API and
+ * running through all classloaders (root, BUNDLEs).
+ *
+ * @ThreadSafe - is immutable
--- End diff --

I agree, and I am working on a refactoring.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132820984
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,436 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * Scans through the classpath to load all extension components using the 
service provider API and
+ * running through all classloaders (root, BUNDLEs).
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  // Maps a service definition (interface) to those classes that implement 
the interface
+  private static final Map definitionMap = new 
HashMap<>();
+
+  private static final Map classNameBundleLookup = 
new HashMap<>();
+  private static final Map 
bundleCoordinateBundleLookup = new HashMap<>();
+  private static final Map classLoaderBundleLookup = 
new HashMap<>();
+
+  private static final Set requiresInstanceClassLoading = new 
HashSet<>();
+  private static final Map instanceClassloaderLookup 
= new ConcurrentHashMap<>();
+
+  private static AtomicBoolean inited = new AtomicBoolean(false);
+
+  // should initialize class definitions
+  public static void initClassDefinitions(final List classes) {
+if (classes != null) {
+  for (Class clazz : classes) {
+definitionMap.put(clazz, new HashSet<>());
+  }
+}
+inited.set(true);
+  }
+
+  public static void resetClassDefinitions() {
+definitionMap.clear();
+inited.set(false);
+  }
+
+  /**
+   * Loads all extension class types that can be found on the bootstrap 
classloader and by creating
+   * classloaders for all BUNDLES found within the classpath.
+   *
+   * @param bundles the bundles to scan through in search of extensions
+   */
+  public static void discoverExtensions(final Bundle systemBundle, final 
Set bundles)
+  throws NotInitializedException {
+checkInitialized();
+// get the current context class loader
+ClassLoader currentContextClassLoader = 
Thread.currentThread().getContextClassLoader();
+
+// load the system bundle first so that any extensions found in JARs 
directly in lib will be registered as
+// being from the system bundle and not from all the other Bundles
+loadExtensions(systemBundle);
+
bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(),
 systemBundle);
+
+// 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132820881
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/VFSBundleClassLoader.java 
---
@@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+import java.io.IOException;
+import java.net.URL;
+import java.security.CodeSource;
+import java.security.Permission;
+import java.security.PermissionCollection;
+import java.security.Permissions;
+import java.security.SecureClassLoader;
+import java.security.cert.Certificate;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Enumeration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.jar.Attributes;
+import java.util.jar.Attributes.Name;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.commons.vfs2.NameScope;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * 
+ * A ClassLoader for loading BUNDLES (plugin archives). BUNDLEs 
are designed to
+ * allow isolating bundles of code (comprising one-or-more
+ * plugin classes and their
+ * dependencies) from other such bundles; this allows for dependencies and
+ * processors that require conflicting, incompatible versions of the same
+ * dependency to run in a single instance of a given process.
+ *
+ * 
+ * BundleClassLoader follows the delegation model described in
+ * {@link ClassLoader#findClass(java.lang.String) 
ClassLoader.findClass(...)};
+ * classes are first loaded from the parent ClassLoader, and only 
if
+ * they cannot be found there does the BundleClassLoader provide a
+ * definition. Specifically, this means that resources are loaded from the 
application's
+ * conf
+ * and lib directories first, and if they cannot be found there, 
are
+ * loaded from the BUNDLE.
+ *
+ * 
+ * The packaging of a BUNDLE is such that it is a ZIP file with the 
following
+ * directory structure:
+ *
+ * 
+ *   +META-INF/
+ *   +-- bundled-dependencies/
+ *   +-- JAR files
+ *   +-- MANIFEST.MF
+ * 
+ * 
+ *
+ * 
+ * The MANIFEST.MF file contains the same information as a typical JAR 
file but
+ * also includes two additional bundle properties: {@code Bundle-Id} and
+ * {@code Bundle-Dependency-Id}.
+ * 
+ *
+ * 
+ * The {@code Bundle-Id} provides a unique identifier for this BUNDLE.
+ * 
+ *
+ * 
+ * The {@code Bundle-Dependency-Id} is optional. If provided, it indicates 
that
+ * this BUNDLE should inherit all of the dependencies of the BUNDLE with 
the provided
+ * ID. Often times, the BUNDLE that is depended upon is referred to as the 
Parent.
+ * This is because its ClassLoader will be the parent ClassLoader of the
+ * dependent BUNDLE.
+ * 
+ *
+ * 
+ * If a BUNDLE is built using the Bundles Maven Plugin, the {@code 
Bundle-Id} property
+ * will be set to the artifactId of the BUNDLE. The {@code 
Bundle-Dependency-Id} will
+ * be set to the artifactId of the BUNDLE that is depended upon. For 
example, if
+ * BUNDLE A is defined as such:
+ *
+ * 
+ * ...
+ * artifactIdbundle-a/artifactId
+ * packagingbundle/packaging
+ * ...
+ * dependencies
+ *   dependency
+ * groupIdgroup/groupId
+ * artifactIdbundle-z/artifactId
+ * typebundle/type
+ *   /dependency
+ * /dependencies
+ * 
+ * 
+ *
+ *
+ * 
+ * Then the MANIFEST.MF file that is created for Bundle A will have the 
following
+ * properties set:
+ * 
+ * {@code {Foo}-Id: bundle-a}
+ * {@code {Foo}-Dependency-Id: bundle-z}
+ * 
+ * Where is configurable by BundleProperty META_ID_PREFIX [ default Bundle 
]
+ * 
+ *
+ * 
+ * Note, 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132808147
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,436 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * Scans through the classpath to load all extension components using the 
service provider API and
+ * running through all classloaders (root, BUNDLEs).
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  // Maps a service definition (interface) to those classes that implement 
the interface
+  private static final Map definitionMap = new 
HashMap<>();
+
+  private static final Map classNameBundleLookup = 
new HashMap<>();
+  private static final Map 
bundleCoordinateBundleLookup = new HashMap<>();
+  private static final Map classLoaderBundleLookup = 
new HashMap<>();
+
+  private static final Set requiresInstanceClassLoading = new 
HashSet<>();
+  private static final Map instanceClassloaderLookup 
= new ConcurrentHashMap<>();
+
+  private static AtomicBoolean inited = new AtomicBoolean(false);
+
+  // should initialize class definitions
+  public static void initClassDefinitions(final List classes) {
+if (classes != null) {
+  for (Class clazz : classes) {
+definitionMap.put(clazz, new HashSet<>());
+  }
+}
+inited.set(true);
+  }
+
+  public static void resetClassDefinitions() {
+definitionMap.clear();
+inited.set(false);
+  }
+
+  /**
+   * Loads all extension class types that can be found on the bootstrap 
classloader and by creating
+   * classloaders for all BUNDLES found within the classpath.
+   *
+   * @param bundles the bundles to scan through in search of extensions
+   */
+  public static void discoverExtensions(final Bundle systemBundle, final 
Set bundles)
+  throws NotInitializedException {
+checkInitialized();
+// get the current context class loader
+ClassLoader currentContextClassLoader = 
Thread.currentThread().getContextClassLoader();
+
+// load the system bundle first so that any extensions found in JARs 
directly in lib will be registered as
+// being from the system bundle and not from all the other Bundles
+loadExtensions(systemBundle);
+
bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(),
 systemBundle);
+
+// 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132808084
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+
+import java.util.*;
+import java.util.function.BiFunction;
+
+public class ExtensionMapping {
+
+  private final Map> 
extensionNameMap = new HashMap<>();
--- End diff --

So - the goal with the Extension* stuff was to remove the necessity to 
statically declare the classes of extensions used.  They are passed in to the 
ExtensionManager and the BundleThreadContextClassLoader.   The ExtensionMapping 
is different, it ends up getting the classes from discover in the nifi 
properties, where the classes are defined, and typed.
So yes, the type does subdivide the namespace.
Type -> Classname -> coordinates
Parser -> MessageParser -> Parser Extension Coordinates



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132801000
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java
 ---
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.util.BundleProperties;
+
+/**
+ * THREAD SAFE
+ */
+public class BundleThreadContextClassLoader extends URLClassLoader {
+
+static final ContextSecurityManager contextSecurityManager = new 
ContextSecurityManager();
+private final ClassLoader forward = ClassLoader.getSystemClassLoader();
+private static final List bundleSpecificClasses = new 
ArrayList<>();
+private static AtomicBoolean inited = new AtomicBoolean(false);
+// should initialize class definitions
+public static void initClasses(final List classes){
+bundleSpecificClasses.clear();
+if(classes != null){
+for( Class clazz : classes){
+bundleSpecificClasses.add(clazz);
+}
+}
+inited.set(true);
+}
+
+public static void resetClasses(){
+bundleSpecificClasses.clear();
+inited.set(false);
+}
+
+private BundleThreadContextClassLoader() {
+super(new URL[0]);
+}
+
+@Override
+public void clearAssertionStatus() {
+lookupClassLoader().clearAssertionStatus();
+}
+
+@Override
+public URL getResource(String name) {
+return lookupClassLoader().getResource(name);
+}
+
+@Override
+public InputStream getResourceAsStream(String name) {
+return lookupClassLoader().getResourceAsStream(name);
+}
+
+@Override
+public Enumeration getResources(String name) throws IOException {
+return lookupClassLoader().getResources(name);
+}
+
+@Override
+public Class loadClass(String name) throws ClassNotFoundException {
+return lookupClassLoader().loadClass(name);
+}
+
+@Override
+public void setClassAssertionStatus(String className, boolean enabled) 
{
+lookupClassLoader().setClassAssertionStatus(className, enabled);
+}
+
+@Override
+public void setDefaultAssertionStatus(boolean enabled) {
+lookupClassLoader().setDefaultAssertionStatus(enabled);
+}
+
+@Override
+public void setPackageAssertionStatus(String packageName, boolean 
enabled) {
+lookupClassLoader().setPackageAssertionStatus(packageName, 
enabled);
+}
+
+private ClassLoader lookupClassLoader() {
+final Class[] classStack = 
contextSecurityManager.getExecutionStack();
+
+for (Class currentClass : classStack) {
+final Class bundleClass = findBundleClass(currentClass);
+if (bundleClass != null) {
+final ClassLoader desiredClassLoader = 
bundleClass.getClassLoader();
+
+// When new Threads are created, the new Thread inherits 
the ClassLoaderContext of
+// the caller. However, the call stack of that new Thread 
may not trace back to any app-specific
+// code. Therefore, the BundleThreadContextClassLoader 
will be unable to find the appropriate Bundle
+// ClassLoader. As a result, we want to set the 
ContextClassLoader to the Bundel 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132799099
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+
+import java.util.*;
+import java.util.function.BiFunction;
+
+public class ExtensionMapping {
+
+  private final Map> 
extensionNameMap = new HashMap<>();
--- End diff --

Nifi had separate namespaces for the different categories of extension.  
This collapses it into one namespace.  That's probably okay, but worth noting.  
Or does the later use of "types" subdivide in a different way? I didn't fully 
follow.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132797485
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,436 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * Scans through the classpath to load all extension components using the 
service provider API and
+ * running through all classloaders (root, BUNDLEs).
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  // Maps a service definition (interface) to those classes that implement 
the interface
+  private static final Map definitionMap = new 
HashMap<>();
+
+  private static final Map classNameBundleLookup = 
new HashMap<>();
+  private static final Map 
bundleCoordinateBundleLookup = new HashMap<>();
+  private static final Map classLoaderBundleLookup = 
new HashMap<>();
+
+  private static final Set requiresInstanceClassLoading = new 
HashSet<>();
+  private static final Map instanceClassloaderLookup 
= new ConcurrentHashMap<>();
+
+  private static AtomicBoolean inited = new AtomicBoolean(false);
+
+  // should initialize class definitions
+  public static void initClassDefinitions(final List classes) {
+if (classes != null) {
+  for (Class clazz : classes) {
+definitionMap.put(clazz, new HashSet<>());
+  }
+}
+inited.set(true);
+  }
+
+  public static void resetClassDefinitions() {
+definitionMap.clear();
+inited.set(false);
+  }
+
+  /**
+   * Loads all extension class types that can be found on the bootstrap 
classloader and by creating
+   * classloaders for all BUNDLES found within the classpath.
+   *
+   * @param bundles the bundles to scan through in search of extensions
+   */
+  public static void discoverExtensions(final Bundle systemBundle, final 
Set bundles)
+  throws NotInitializedException {
+checkInitialized();
+// get the current context class loader
+ClassLoader currentContextClassLoader = 
Thread.currentThread().getContextClassLoader();
+
+// load the system bundle first so that any extensions found in JARs 
directly in lib will be registered as
+// being from the system bundle and not from all the other Bundles
+loadExtensions(systemBundle);
+
bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(),
 systemBundle);
+
+// 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132801955
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/VFSBundleClassLoader.java 
---
@@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+import java.io.IOException;
+import java.net.URL;
+import java.security.CodeSource;
+import java.security.Permission;
+import java.security.PermissionCollection;
+import java.security.Permissions;
+import java.security.SecureClassLoader;
+import java.security.cert.Certificate;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Enumeration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.jar.Attributes;
+import java.util.jar.Attributes.Name;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.commons.vfs2.NameScope;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * 
+ * A ClassLoader for loading BUNDLES (plugin archives). BUNDLEs 
are designed to
+ * allow isolating bundles of code (comprising one-or-more
+ * plugin classes and their
+ * dependencies) from other such bundles; this allows for dependencies and
+ * processors that require conflicting, incompatible versions of the same
+ * dependency to run in a single instance of a given process.
+ *
+ * 
+ * BundleClassLoader follows the delegation model described in
+ * {@link ClassLoader#findClass(java.lang.String) 
ClassLoader.findClass(...)};
+ * classes are first loaded from the parent ClassLoader, and only 
if
+ * they cannot be found there does the BundleClassLoader provide a
+ * definition. Specifically, this means that resources are loaded from the 
application's
+ * conf
+ * and lib directories first, and if they cannot be found there, 
are
+ * loaded from the BUNDLE.
+ *
+ * 
+ * The packaging of a BUNDLE is such that it is a ZIP file with the 
following
+ * directory structure:
+ *
+ * 
+ *   +META-INF/
+ *   +-- bundled-dependencies/
+ *   +-- JAR files
+ *   +-- MANIFEST.MF
+ * 
+ * 
+ *
+ * 
+ * The MANIFEST.MF file contains the same information as a typical JAR 
file but
+ * also includes two additional bundle properties: {@code Bundle-Id} and
+ * {@code Bundle-Dependency-Id}.
+ * 
+ *
+ * 
+ * The {@code Bundle-Id} provides a unique identifier for this BUNDLE.
+ * 
+ *
+ * 
+ * The {@code Bundle-Dependency-Id} is optional. If provided, it indicates 
that
+ * this BUNDLE should inherit all of the dependencies of the BUNDLE with 
the provided
+ * ID. Often times, the BUNDLE that is depended upon is referred to as the 
Parent.
+ * This is because its ClassLoader will be the parent ClassLoader of the
+ * dependent BUNDLE.
+ * 
+ *
+ * 
+ * If a BUNDLE is built using the Bundles Maven Plugin, the {@code 
Bundle-Id} property
+ * will be set to the artifactId of the BUNDLE. The {@code 
Bundle-Dependency-Id} will
+ * be set to the artifactId of the BUNDLE that is depended upon. For 
example, if
+ * BUNDLE A is defined as such:
+ *
+ * 
+ * ...
+ * artifactIdbundle-a/artifactId
+ * packagingbundle/packaging
+ * ...
+ * dependencies
+ *   dependency
+ * groupIdgroup/groupId
+ * artifactIdbundle-z/artifactId
+ * typebundle/type
+ *   /dependency
+ * /dependencies
+ * 
+ * 
+ *
+ *
+ * 
+ * Then the MANIFEST.MF file that is created for Bundle A will have the 
following
+ * properties set:
+ * 
+ * {@code {Foo}-Id: bundle-a}
+ * {@code {Foo}-Dependency-Id: bundle-z}
+ * 
+ * Where is configurable by BundleProperty META_ID_PREFIX [ default Bundle 
]
+ * 
+ *
+ * 
+ * Note, 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132795983
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,436 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * Scans through the classpath to load all extension components using the 
service provider API and
+ * running through all classloaders (root, BUNDLEs).
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  // Maps a service definition (interface) to those classes that implement 
the interface
+  private static final Map definitionMap = new 
HashMap<>();
+
+  private static final Map classNameBundleLookup = 
new HashMap<>();
+  private static final Map 
bundleCoordinateBundleLookup = new HashMap<>();
+  private static final Map classLoaderBundleLookup = 
new HashMap<>();
+
+  private static final Set requiresInstanceClassLoading = new 
HashSet<>();
+  private static final Map instanceClassloaderLookup 
= new ConcurrentHashMap<>();
+
+  private static AtomicBoolean inited = new AtomicBoolean(false);
+
+  // should initialize class definitions
+  public static void initClassDefinitions(final List classes) {
+if (classes != null) {
+  for (Class clazz : classes) {
+definitionMap.put(clazz, new HashSet<>());
+  }
+}
+inited.set(true);
+  }
+
+  public static void resetClassDefinitions() {
+definitionMap.clear();
+inited.set(false);
+  }
+
+  /**
+   * Loads all extension class types that can be found on the bootstrap 
classloader and by creating
+   * classloaders for all BUNDLES found within the classpath.
+   *
+   * @param bundles the bundles to scan through in search of extensions
+   */
+  public static void discoverExtensions(final Bundle systemBundle, final 
Set bundles)
+  throws NotInitializedException {
+checkInitialized();
+// get the current context class loader
+ClassLoader currentContextClassLoader = 
Thread.currentThread().getContextClassLoader();
+
+// load the system bundle first so that any extensions found in JARs 
directly in lib will be registered as
+// being from the system bundle and not from all the other Bundles
+loadExtensions(systemBundle);
+
bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(),
 systemBundle);
+
+// 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132801851
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/VFSBundleClassLoader.java 
---
@@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+import java.io.IOException;
+import java.net.URL;
+import java.security.CodeSource;
+import java.security.Permission;
+import java.security.PermissionCollection;
+import java.security.Permissions;
+import java.security.SecureClassLoader;
+import java.security.cert.Certificate;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Enumeration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.jar.Attributes;
+import java.util.jar.Attributes.Name;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.commons.vfs2.NameScope;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * 
+ * A ClassLoader for loading BUNDLES (plugin archives). BUNDLEs 
are designed to
+ * allow isolating bundles of code (comprising one-or-more
+ * plugin classes and their
+ * dependencies) from other such bundles; this allows for dependencies and
+ * processors that require conflicting, incompatible versions of the same
+ * dependency to run in a single instance of a given process.
+ *
+ * 
+ * BundleClassLoader follows the delegation model described in
+ * {@link ClassLoader#findClass(java.lang.String) 
ClassLoader.findClass(...)};
+ * classes are first loaded from the parent ClassLoader, and only 
if
+ * they cannot be found there does the BundleClassLoader provide a
+ * definition. Specifically, this means that resources are loaded from the 
application's
+ * conf
+ * and lib directories first, and if they cannot be found there, 
are
+ * loaded from the BUNDLE.
+ *
+ * 
+ * The packaging of a BUNDLE is such that it is a ZIP file with the 
following
+ * directory structure:
+ *
+ * 
+ *   +META-INF/
+ *   +-- bundled-dependencies/
+ *   +-- JAR files
+ *   +-- MANIFEST.MF
+ * 
+ * 
+ *
+ * 
+ * The MANIFEST.MF file contains the same information as a typical JAR 
file but
+ * also includes two additional bundle properties: {@code Bundle-Id} and
+ * {@code Bundle-Dependency-Id}.
+ * 
+ *
+ * 
+ * The {@code Bundle-Id} provides a unique identifier for this BUNDLE.
+ * 
+ *
+ * 
+ * The {@code Bundle-Dependency-Id} is optional. If provided, it indicates 
that
+ * this BUNDLE should inherit all of the dependencies of the BUNDLE with 
the provided
+ * ID. Often times, the BUNDLE that is depended upon is referred to as the 
Parent.
+ * This is because its ClassLoader will be the parent ClassLoader of the
+ * dependent BUNDLE.
+ * 
+ *
+ * 
+ * If a BUNDLE is built using the Bundles Maven Plugin, the {@code 
Bundle-Id} property
+ * will be set to the artifactId of the BUNDLE. The {@code 
Bundle-Dependency-Id} will
+ * be set to the artifactId of the BUNDLE that is depended upon. For 
example, if
+ * BUNDLE A is defined as such:
+ *
+ * 
+ * ...
+ * artifactIdbundle-a/artifactId
+ * packagingbundle/packaging
+ * ...
+ * dependencies
+ *   dependency
+ * groupIdgroup/groupId
+ * artifactIdbundle-z/artifactId
+ * typebundle/type
+ *   /dependency
+ * /dependencies
+ * 
+ * 
+ *
+ *
+ * 
+ * Then the MANIFEST.MF file that is created for Bundle A will have the 
following
+ * properties set:
+ * 
+ * {@code {Foo}-Id: bundle-a}
+ * {@code {Foo}-Dependency-Id: bundle-z}
+ * 
+ * Where is configurable by BundleProperty META_ID_PREFIX [ default Bundle 
]
+ * 
+ *
+ * 
+ * Note, 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132796537
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,436 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import 
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * Scans through the classpath to load all extension components using the 
service provider API and
+ * running through all classloaders (root, BUNDLEs).
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(ExtensionManager.class);
+
+  public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new 
BundleCoordinates(
+  BundleCoordinates.DEFAULT_GROUP, "system", 
BundleCoordinates.DEFAULT_VERSION);
+
+  // Maps a service definition (interface) to those classes that implement 
the interface
+  private static final Map definitionMap = new 
HashMap<>();
+
+  private static final Map classNameBundleLookup = 
new HashMap<>();
+  private static final Map 
bundleCoordinateBundleLookup = new HashMap<>();
+  private static final Map classLoaderBundleLookup = 
new HashMap<>();
+
+  private static final Set requiresInstanceClassLoading = new 
HashSet<>();
+  private static final Map instanceClassloaderLookup 
= new ConcurrentHashMap<>();
+
+  private static AtomicBoolean inited = new AtomicBoolean(false);
+
+  // should initialize class definitions
+  public static void initClassDefinitions(final List classes) {
+if (classes != null) {
+  for (Class clazz : classes) {
+definitionMap.put(clazz, new HashSet<>());
+  }
+}
+inited.set(true);
+  }
+
+  public static void resetClassDefinitions() {
+definitionMap.clear();
+inited.set(false);
+  }
+
+  /**
+   * Loads all extension class types that can be found on the bootstrap 
classloader and by creating
+   * classloaders for all BUNDLES found within the classpath.
+   *
+   * @param bundles the bundles to scan through in search of extensions
+   */
+  public static void discoverExtensions(final Bundle systemBundle, final 
Set bundles)
+  throws NotInitializedException {
+checkInitialized();
+// get the current context class loader
+ClassLoader currentContextClassLoader = 
Thread.currentThread().getContextClassLoader();
+
+// load the system bundle first so that any extensions found in JARs 
directly in lib will be registered as
+// being from the system bundle and not from all the other Bundles
+loadExtensions(systemBundle);
+
bundleCoordinateBundleLookup.put(systemBundle.getBundleDetails().getCoordinates(),
 systemBundle);
+
+// 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132284538
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/bundle/BundleDetails.java 
---
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.bundle;
+
+import org.apache.commons.vfs2.FileObject;
+
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+
+/**
+ * Metadata about a bundle.
+ * the withCoordinates and withBundleFile properties are required
+ */
+public class BundleDetails {
+
+private final FileObject bundleFile;
+
+private final BundleCoordinates coordinates;
+private final BundleCoordinates dependencyCoordinates;
+
+private final String buildTag;
+private final String buildRevision;
+private final String buildBranch;
+private final String buildTimestamp;
+private final String buildJdk;
+private final String builtBy;
+
+private BundleDetails(final Builder builder) {
+this.bundleFile = builder.bundleFile;
+this.coordinates = builder.coordinates;
+this.dependencyCoordinates = builder.dependencyCoordinates;
+
+this.buildTag = builder.buildTag;
+this.buildRevision = builder.buildRevision;
+this.buildBranch = builder.buildBranch;
+this.buildTimestamp = builder.buildTimestamp;
+this.buildJdk = builder.buildJdk;
+this.builtBy = builder.builtBy;
+
+if (this.coordinates == null) {
+if (this.bundleFile == null) {
+throw new IllegalStateException("Coordinate cannot be 
null");
+} else {
+throw new IllegalStateException("Coordinate cannot be null 
for " + this.bundleFile.getName());
+}
+}
+
+if (this.bundleFile == null) {
+throw new IllegalStateException("bundleFile cannot be null for 
" + this.coordinates
+.getId());
+}
+}
+
+public FileObject getBundleFile() {
+return bundleFile;
+}
+
+public BundleCoordinates getCoordinates() {
+return coordinates;
+}
+
+public BundleCoordinates getDependencyCoordinates() {
+return dependencyCoordinates;
+}
+
+public String getBuildTag() {
+return buildTag;
+}
+
+public String getBuildRevision() {
+return buildRevision;
+}
+
+public String getBuildBranch() {
+return buildBranch;
+}
+
+public String getBuildTimestamp() {
+return buildTimestamp;
+}
+
+public String getBuildJdk() {
+return buildJdk;
+}
+
+public String getBuiltBy() {
+return builtBy;
+}
+
+@Override
+public String toString() {
+return coordinates.toString();
+}
+
+public Date getBuildTimestampDate() {
+if (buildTimestamp != null && !buildTimestamp.isEmpty()) {
+try {
+SimpleDateFormat buildTimestampFormat = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss'Z'");
+Date buildTimestampDate = 
buildTimestampFormat.parse(buildTimestamp);
+return buildTimestampDate;
+} catch (ParseException parseEx) {
+return null;
+}
+} else {
+return null;
+}
+}
+
+/**
+ * Builder for BundleDetails.
--- End diff --

done.  
I also did the validation you had mentioned.  When I got in there it made 
more sense than not.  Sorry I didn't see it before


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132280796
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/bundle/BundleCoordinates.java
 ---
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.bundle;
+
+/**
+ * The coordinates of a bundle (group, artifact, version).
+ */
+public class BundleCoordinates {
+
+public static final String DEFAULT_GROUP = "default";
+public static final String DEFAULT_VERSION = "unversioned";
+
+public static final BundleCoordinates UNKNOWN_COORDINATE = new 
BundleCoordinates(DEFAULT_GROUP, "unknown", DEFAULT_VERSION);
+
+private final String group;
+private final String id;
+private final String version;
+private final String coordinate;
+
+public BundleCoordinates(final String group, final String id, final 
String version) {
+this.group = isBlank(group) ? DEFAULT_GROUP : group;
+this.id = id;
+this.version = isBlank(version) ? DEFAULT_VERSION : version;
+
+if (isBlank(id)) {
+throw new IllegalStateException("Id is required for 
BundleCoordinates");
+}
+
+if(this.group.contains(":") || this.id.contains(":") || 
this.version.contains(":")) {
+throw new IllegalStateException(String.format("Invalid 
coordinates: cannot contain : character group[%s] id[%s] 
version[%s]",this.group,this.id,this.version));
+}
+this.coordinate = this.group + ":" + this.id + ":" + this.version;
+}
+
+private boolean isBlank(String str) {
+return str == null || str.trim().length() == 0;
+}
+
+public String getGroup() {
+return group;
+}
+
+public String getId() {
+return id;
+}
+
+public String getVersion() {
+return version;
+}
+
+public final String getCoordinate() {
--- End diff --

Please rename this to getCoordinates() too, since it is public.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132279911
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/bundle/BundleDetails.java 
---
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.bundle;
+
+import org.apache.commons.vfs2.FileObject;
+
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+
+/**
+ * Metadata about a bundle.
+ * the withCoordinates and withBundleFile properties are required
+ */
+public class BundleDetails {
+
+private final FileObject bundleFile;
+
+private final BundleCoordinates coordinates;
+private final BundleCoordinates dependencyCoordinates;
+
+private final String buildTag;
+private final String buildRevision;
+private final String buildBranch;
+private final String buildTimestamp;
+private final String buildJdk;
+private final String builtBy;
+
+private BundleDetails(final Builder builder) {
+this.bundleFile = builder.bundleFile;
+this.coordinates = builder.coordinates;
+this.dependencyCoordinates = builder.dependencyCoordinates;
+
+this.buildTag = builder.buildTag;
+this.buildRevision = builder.buildRevision;
+this.buildBranch = builder.buildBranch;
+this.buildTimestamp = builder.buildTimestamp;
+this.buildJdk = builder.buildJdk;
+this.builtBy = builder.builtBy;
+
+if (this.coordinates == null) {
+if (this.bundleFile == null) {
+throw new IllegalStateException("Coordinate cannot be 
null");
+} else {
+throw new IllegalStateException("Coordinate cannot be null 
for " + this.bundleFile.getName());
+}
+}
+
+if (this.bundleFile == null) {
+throw new IllegalStateException("bundleFile cannot be null for 
" + this.coordinates
+.getId());
+}
+}
+
+public FileObject getBundleFile() {
+return bundleFile;
+}
+
+public BundleCoordinates getCoordinates() {
+return coordinates;
+}
+
+public BundleCoordinates getDependencyCoordinates() {
+return dependencyCoordinates;
+}
+
+public String getBuildTag() {
+return buildTag;
+}
+
+public String getBuildRevision() {
+return buildRevision;
+}
+
+public String getBuildBranch() {
+return buildBranch;
+}
+
+public String getBuildTimestamp() {
+return buildTimestamp;
+}
+
+public String getBuildJdk() {
+return buildJdk;
+}
+
+public String getBuiltBy() {
+return builtBy;
+}
+
+@Override
+public String toString() {
+return coordinates.toString();
+}
+
+public Date getBuildTimestampDate() {
+if (buildTimestamp != null && !buildTimestamp.isEmpty()) {
+try {
+SimpleDateFormat buildTimestampFormat = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss'Z'");
+Date buildTimestampDate = 
buildTimestampFormat.parse(buildTimestamp);
+return buildTimestampDate;
+} catch (ParseException parseEx) {
+return null;
+}
+} else {
+return null;
+}
+}
+
+/**
+ * Builder for BundleDetails.
--- End diff --

Please also javadoc the required properties on the Builder, since that the 
public i/f that actually gets used.
> + * the withCoordinates and withBundleFile properties are required


---
If your project is set up for it, you can reply to this email and have your
reply 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132263102
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132261444
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
--- End diff --

But now I'm also concerned, if your design point was single extensions 
only:  If I add multiple extensions, will they all be loaded and available?  It 
looked to me like they would, but maybe not if you were specifically thinking 
otherwise?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132258789
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/bundle/BundleDetails.java 
---
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.bundle;
+
+import org.apache.commons.vfs2.FileObject;
+
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+
+/**
+ * Metadata about a bundle.
+ */
+public class BundleDetails {
+
+private final FileObject bundleFile;
+
+private final BundleCoordinate coordinate;
+private final BundleCoordinate dependencyCoordinate;
+
+private final String buildTag;
+private final String buildRevision;
+private final String buildBranch;
+private final String buildTimestamp;
+private final String buildJdk;
+private final String builtBy;
+
+private BundleDetails(final Builder builder) {
+this.bundleFile = builder.bundleFile;
+this.coordinate = builder.coordinate;
+this.dependencyCoordinate = builder.dependencyCoordinate;
+
+this.buildTag = builder.buildTag;
+this.buildRevision = builder.buildRevision;
+this.buildBranch = builder.buildBranch;
+this.buildTimestamp = builder.buildTimestamp;
+this.buildJdk = builder.buildJdk;
+this.builtBy = builder.builtBy;
+
+if (this.coordinate == null) {
+if (this.bundleFile == null) {
+throw new IllegalStateException("Coordinate cannot be 
null");
+} else {
+throw new IllegalStateException("Coordinate cannot be null 
for " + this.bundleFile.getName());
+}
+}
+
+if (this.bundleFile == null) {
+throw new IllegalStateException("Working directory cannot be 
null for " + this.coordinate.getId());
+}
+}
+
+public FileObject getBundleFile() {
+return bundleFile;
+}
+
+public BundleCoordinate getCoordinate() {
+return coordinate;
+}
+
+public BundleCoordinate getDependencyCoordinate() {
+return dependencyCoordinate;
+}
+
+public String getBuildTag() {
+return buildTag;
+}
+
+public String getBuildRevision() {
+return buildRevision;
+}
+
+public String getBuildBranch() {
+return buildBranch;
+}
+
+public String getBuildTimestamp() {
+return buildTimestamp;
+}
+
+public String getBuildJdk() {
+return buildJdk;
+}
+
+public String getBuiltBy() {
+return builtBy;
+}
+
+@Override
+public String toString() {
+return coordinate.toString();
+}
+
+public Date getBuildTimestampDate() {
+if (buildTimestamp != null && !buildTimestamp.isEmpty()) {
+try {
+SimpleDateFormat buildTimestampFormat = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss'Z'");
+Date buildTimestampDate = 
buildTimestampFormat.parse(buildTimestamp);
+return buildTimestampDate;
+} catch (ParseException parseEx) {
+return null;
+}
+} else {
+return null;
+}
+}
+
+/**
+ * Builder for BundleDetails.
+ */
+public static class Builder {
+
+private FileObject bundleFile;
+
+private BundleCoordinate coordinate;
+private BundleCoordinate dependencyCoordinate;
+
+private String buildTag;
+private String buildRevision;
+private String buildBranch;
+private String buildTimestamp;
+

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132255328
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/bundle/Bundle.java ---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.bundle;
+
+/**
+ * Represents a bundle that contains one or more extensions.
+ */
+public class Bundle {
+
+private final BundleDetails bundleDetails;
+
+private final ClassLoader classLoader;
+
+public Bundle(final BundleDetails bundleDetails, final ClassLoader 
classLoader) {
+this.bundleDetails = bundleDetails;
+this.classLoader = classLoader;
+
+if (this.bundleDetails == null) {
+throw new IllegalStateException("BundleDetails cannot be 
null");
+}
+
+if (this.classLoader == null) {
+throw new IllegalStateException("ClassLoader cannot be null");
--- End diff --

@ottobackwards , regarding
> That and it is a runtime exception?

Both IllegalStateException and IllegalArgumentException are 
RuntimeExceptions, hence unchecked.  My comment "It is also a RuntimeException" 
was intended to say "It is easy to substitute IllegalArgumentException here, 
because it doesn't change the exception-checking behavior."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132254112
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/util/FileSystemManagerFactory.java
 ---
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.util;
+
+import org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator;
+import org.apache.commons.vfs2.*;
+import org.apache.commons.vfs2.cache.SoftRefFilesCache;
+import org.apache.commons.vfs2.impl.DefaultFileSystemManager;
+import org.apache.commons.vfs2.impl.FileContentInfoFilenameFactory;
+import org.apache.commons.vfs2.provider.hdfs.HdfsFileProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+
+public class FileSystemManagerFactory {
+  private static final Logger LOG = 
LoggerFactory.getLogger(FileSystemManagerFactory.class);
+
+  /**
+   * Create a FileSystem manager suitable for our purposes.
+   * This manager supports files of the following types:
+   * * res - resource files
+   * * jar
+   * * tar
+   * * bz2
+   * * tgz
+   * * zip
+   * * HDFS
+   * * FTP
+   * * HTTP/S
+   * * file
+   * @return
+   * @throws FileSystemException
+   */
+  public static FileSystemManager createFileSystemManager() throws 
FileSystemException {
+return createFileSystemManager(null);
+  }
+  public static FileSystemManager createFileSystemManager(String 
jarExtensionToRegister) throws FileSystemException {
+DefaultFileSystemManager vfs = new DefaultFileSystemManager();
+
+if(jarExtensionToRegister != null || 
!StringUtils.isBlank(jarExtensionToRegister) ) {
+  vfs.addExtensionMap(jarExtensionToRegister, "jar");
+  vfs.addProvider(jarExtensionToRegister, new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+}
+
+vfs.addProvider("res", new 
org.apache.commons.vfs2.provider.res.ResourceFileProvider());
+vfs.addProvider("zip", new 
org.apache.commons.vfs2.provider.zip.ZipFileProvider());
+vfs.addProvider("gz", new 
org.apache.commons.vfs2.provider.gzip.GzipFileProvider());
+vfs.addProvider("ram", new 
org.apache.commons.vfs2.provider.ram.RamFileProvider());
+vfs.addProvider("file", new 
org.apache.commons.vfs2.provider.local.DefaultLocalFileProvider());
+vfs.addProvider("jar", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("http", new 
org.apache.commons.vfs2.provider.http.HttpFileProvider());
+vfs.addProvider("https", new 
org.apache.commons.vfs2.provider.https.HttpsFileProvider());
+vfs.addProvider("ftp", new 
org.apache.commons.vfs2.provider.ftp.FtpFileProvider());
+vfs.addProvider("ftps", new 
org.apache.commons.vfs2.provider.ftps.FtpsFileProvider());
+vfs.addProvider("war", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("par", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("ear", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("sar", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("ejb3", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("tmp", new 
org.apache.commons.vfs2.provider.temp.TemporaryFileProvider());
+vfs.addProvider("tar", new 
org.apache.commons.vfs2.provider.tar.TarFileProvider());
+vfs.addProvider("tbz2", new 
org.apache.commons.vfs2.provider.tar.TarFileProvider());
+vfs.addProvider("tgz", new 
org.apache.commons.vfs2.provider.tar.TarFileProvider());
+vfs.addProvider("bz2", new 
org.apache.commons.vfs2.provider.bzip2.Bzip2FileProvider());
+vfs.addProvider("hdfs", new HdfsFileProvider());
+vfs.addExtensionMap("jar", "jar");
+vfs.addExtensionMap("zip", "zip");
+vfs.addExtensionMap("gz", "gz");
+

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r132012486
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/bundle/Bundle.java ---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.bundle;
+
+/**
+ * Represents a bundle that contains one or more extensions.
+ */
+public class Bundle {
+
+private final BundleDetails bundleDetails;
+
+private final ClassLoader classLoader;
+
+public Bundle(final BundleDetails bundleDetails, final ClassLoader 
classLoader) {
+this.bundleDetails = bundleDetails;
+this.classLoader = classLoader;
+
+if (this.bundleDetails == null) {
+throw new IllegalStateException("BundleDetails cannot be 
null");
+}
+
+if (this.classLoader == null) {
+throw new IllegalStateException("ClassLoader cannot be null");
--- End diff --

No worries, sounds good. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131999633
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/bundle/Bundle.java ---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.bundle;
+
+/**
+ * Represents a bundle that contains one or more extensions.
+ */
+public class Bundle {
+
+private final BundleDetails bundleDetails;
+
+private final ClassLoader classLoader;
+
+public Bundle(final BundleDetails bundleDetails, final ClassLoader 
classLoader) {
+this.bundleDetails = bundleDetails;
+this.classLoader = classLoader;
+
+if (this.bundleDetails == null) {
+throw new IllegalStateException("BundleDetails cannot be 
null");
+}
+
+if (this.classLoader == null) {
+throw new IllegalStateException("ClassLoader cannot be null");
--- End diff --

@bbende sorry to pull you into the middle of this opus.  I was planning on 
talking to you about this after it passed review and settled a bit.

I *think* there a couple of things in here that may be worth a thought for 
nifi, and my plan is to bring that up later.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131995115
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131994814
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131991006
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
--- End diff --

No, it is I who doesn't understand.  I was not thinking about the 
supervisor correctly.  So, adding extensions without restart needs more thought.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131986154
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/bundle/Bundle.java ---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.bundle;
+
+/**
+ * Represents a bundle that contains one or more extensions.
+ */
+public class Bundle {
+
+private final BundleDetails bundleDetails;
+
+private final ClassLoader classLoader;
+
+public Bundle(final BundleDetails bundleDetails, final ClassLoader 
classLoader) {
+this.bundleDetails = bundleDetails;
+this.classLoader = classLoader;
+
+if (this.bundleDetails == null) {
+throw new IllegalStateException("BundleDetails cannot be 
null");
+}
+
+if (this.classLoader == null) {
+throw new IllegalStateException("ClassLoader cannot be null");
--- End diff --

The overall goal was just to ensure that if any code later on encounters a 
Bundle, it can be sure that the details and class loader are present. There 
isn't a realistic case in NiFi where you should encounter these exceptions, it 
was just being defensive.

I think I got in the habit of using IllegalStateException here since at 
this point you are checking the member variables of the object and saying that 
something in the state of the object is not valid, as opposed to checking the 
arguments before assigning them which would maybe better to use 
IllegalArgumentException. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131984525
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/bundle/BundleDetails.java 
---
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.bundle;
+
+import org.apache.commons.vfs2.FileObject;
+
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+
+/**
+ * Metadata about a bundle.
+ */
+public class BundleDetails {
+
+private final FileObject bundleFile;
+
+private final BundleCoordinate coordinate;
+private final BundleCoordinate dependencyCoordinate;
+
+private final String buildTag;
+private final String buildRevision;
+private final String buildBranch;
+private final String buildTimestamp;
+private final String buildJdk;
+private final String builtBy;
+
+private BundleDetails(final Builder builder) {
+this.bundleFile = builder.bundleFile;
+this.coordinate = builder.coordinate;
+this.dependencyCoordinate = builder.dependencyCoordinate;
+
+this.buildTag = builder.buildTag;
+this.buildRevision = builder.buildRevision;
+this.buildBranch = builder.buildBranch;
+this.buildTimestamp = builder.buildTimestamp;
+this.buildJdk = builder.buildJdk;
+this.builtBy = builder.builtBy;
+
+if (this.coordinate == null) {
+if (this.bundleFile == null) {
+throw new IllegalStateException("Coordinate cannot be 
null");
+} else {
+throw new IllegalStateException("Coordinate cannot be null 
for " + this.bundleFile.getName());
+}
+}
+
+if (this.bundleFile == null) {
+throw new IllegalStateException("Working directory cannot be 
null for " + this.coordinate.getId());
+}
+}
+
+public FileObject getBundleFile() {
+return bundleFile;
+}
+
+public BundleCoordinate getCoordinate() {
+return coordinate;
+}
+
+public BundleCoordinate getDependencyCoordinate() {
+return dependencyCoordinate;
+}
+
+public String getBuildTag() {
+return buildTag;
+}
+
+public String getBuildRevision() {
+return buildRevision;
+}
+
+public String getBuildBranch() {
+return buildBranch;
+}
+
+public String getBuildTimestamp() {
+return buildTimestamp;
+}
+
+public String getBuildJdk() {
+return buildJdk;
+}
+
+public String getBuiltBy() {
+return builtBy;
+}
+
+@Override
+public String toString() {
+return coordinate.toString();
+}
+
+public Date getBuildTimestampDate() {
+if (buildTimestamp != null && !buildTimestamp.isEmpty()) {
+try {
+SimpleDateFormat buildTimestampFormat = new 
SimpleDateFormat("-MM-dd'T'HH:mm:ss'Z'");
+Date buildTimestampDate = 
buildTimestampFormat.parse(buildTimestamp);
+return buildTimestampDate;
+} catch (ParseException parseEx) {
+return null;
+}
+} else {
+return null;
+}
+}
+
+/**
+ * Builder for BundleDetails.
+ */
+public static class Builder {
+
+private FileObject bundleFile;
+
+private BundleCoordinate coordinate;
+private BundleCoordinate dependencyCoordinate;
+
+private String buildTag;
+private String buildRevision;
+private String buildBranch;
+private String buildTimestamp;
+

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131983133
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/bundle/BundleDetails.java 
---
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.bundle;
+
+import org.apache.commons.vfs2.FileObject;
+
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+
+/**
+ * Metadata about a bundle.
+ */
+public class BundleDetails {
+
+private final FileObject bundleFile;
+
+private final BundleCoordinate coordinate;
+private final BundleCoordinate dependencyCoordinate;
+
+private final String buildTag;
+private final String buildRevision;
+private final String buildBranch;
+private final String buildTimestamp;
+private final String buildJdk;
+private final String builtBy;
+
+private BundleDetails(final Builder builder) {
+this.bundleFile = builder.bundleFile;
+this.coordinate = builder.coordinate;
+this.dependencyCoordinate = builder.dependencyCoordinate;
+
+this.buildTag = builder.buildTag;
+this.buildRevision = builder.buildRevision;
+this.buildBranch = builder.buildBranch;
+this.buildTimestamp = builder.buildTimestamp;
+this.buildJdk = builder.buildJdk;
+this.builtBy = builder.builtBy;
+
+if (this.coordinate == null) {
+if (this.bundleFile == null) {
+throw new IllegalStateException("Coordinate cannot be 
null");
+} else {
+throw new IllegalStateException("Coordinate cannot be null 
for " + this.bundleFile.getName());
+}
+}
+
+if (this.bundleFile == null) {
+throw new IllegalStateException("Working directory cannot be 
null for " + this.coordinate.getId());
+}
--- End diff --

I think in this context this is enough


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131982207
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/bundle/BundleCoordinate.java
 ---
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.bundle;
+
+/**
+ * The coordinates of a bundle (group, artifact, version).
+ */
+public class BundleCoordinate {
+
+public static final String DEFAULT_GROUP = "default";
+public static final String DEFAULT_VERSION = "unversioned";
+
+public static final BundleCoordinate UNKNOWN_COORDINATE = new 
BundleCoordinate(DEFAULT_GROUP, "unknown", DEFAULT_VERSION);
+
+private final String group;
+private final String id;
+private final String version;
+private final String coordinate;
+
+public BundleCoordinate(final String group, final String id, final 
String version) {
--- End diff --

sure


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131979119
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/util/FileSystemManagerFactory.java
 ---
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.util;
+
+import org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator;
+import org.apache.commons.vfs2.*;
+import org.apache.commons.vfs2.cache.SoftRefFilesCache;
+import org.apache.commons.vfs2.impl.DefaultFileSystemManager;
+import org.apache.commons.vfs2.impl.FileContentInfoFilenameFactory;
+import org.apache.commons.vfs2.provider.hdfs.HdfsFileProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+
+public class FileSystemManagerFactory {
+  private static final Logger LOG = 
LoggerFactory.getLogger(FileSystemManagerFactory.class);
+
+  /**
+   * Create a FileSystem manager suitable for our purposes.
+   * This manager supports files of the following types:
+   * * res - resource files
+   * * jar
+   * * tar
+   * * bz2
+   * * tgz
+   * * zip
+   * * HDFS
+   * * FTP
+   * * HTTP/S
+   * * file
+   * @return
+   * @throws FileSystemException
+   */
+  public static FileSystemManager createFileSystemManager() throws 
FileSystemException {
+return createFileSystemManager(null);
+  }
+  public static FileSystemManager createFileSystemManager(String 
jarExtensionToRegister) throws FileSystemException {
+DefaultFileSystemManager vfs = new DefaultFileSystemManager();
+
+if(jarExtensionToRegister != null || 
!StringUtils.isBlank(jarExtensionToRegister) ) {
+  vfs.addExtensionMap(jarExtensionToRegister, "jar");
+  vfs.addProvider(jarExtensionToRegister, new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+}
+
+vfs.addProvider("res", new 
org.apache.commons.vfs2.provider.res.ResourceFileProvider());
+vfs.addProvider("zip", new 
org.apache.commons.vfs2.provider.zip.ZipFileProvider());
+vfs.addProvider("gz", new 
org.apache.commons.vfs2.provider.gzip.GzipFileProvider());
+vfs.addProvider("ram", new 
org.apache.commons.vfs2.provider.ram.RamFileProvider());
+vfs.addProvider("file", new 
org.apache.commons.vfs2.provider.local.DefaultLocalFileProvider());
+vfs.addProvider("jar", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("http", new 
org.apache.commons.vfs2.provider.http.HttpFileProvider());
+vfs.addProvider("https", new 
org.apache.commons.vfs2.provider.https.HttpsFileProvider());
+vfs.addProvider("ftp", new 
org.apache.commons.vfs2.provider.ftp.FtpFileProvider());
+vfs.addProvider("ftps", new 
org.apache.commons.vfs2.provider.ftps.FtpsFileProvider());
+vfs.addProvider("war", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("par", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("ear", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("sar", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("ejb3", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("tmp", new 
org.apache.commons.vfs2.provider.temp.TemporaryFileProvider());
+vfs.addProvider("tar", new 
org.apache.commons.vfs2.provider.tar.TarFileProvider());
+vfs.addProvider("tbz2", new 
org.apache.commons.vfs2.provider.tar.TarFileProvider());
+vfs.addProvider("tgz", new 
org.apache.commons.vfs2.provider.tar.TarFileProvider());
+vfs.addProvider("bz2", new 
org.apache.commons.vfs2.provider.bzip2.Bzip2FileProvider());
+vfs.addProvider("hdfs", new HdfsFileProvider());
+vfs.addExtensionMap("jar", "jar");
+vfs.addExtensionMap("zip", "zip");
+vfs.addExtensionMap("gz", "gz");
+

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131974209
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131973071
  
--- Diff: 
metron-maven-archetypes/metron-maven-parser-extension-archetype/pom.xml ---
@@ -0,0 +1,46 @@
+
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+  4.0.0
+
+  org.apache.metron
+  metron-maven-parser-extension-archetype
+  0.4.1
+  maven-archetype
+
+  metron-maven-parser-extension-archetype
+
+  
+
+  
+org.apache.maven.archetype
+archetype-packaging
+3.0.0
+  
+
+
+
+  
+
+  maven-archetype-plugin
+  3.0.1
+
--- End diff --

Yes it generates a bundle project for each parser extension.
The parsers execute the 'bundle' phase and build the bundles.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131968916
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131968461
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131967849
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131967527
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131966882
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131965922
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131964579
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
--- End diff --

Are we confident that such resets in a "real" system will release all the 
built-up classloaders and their extensive data structures? ie, nobody elsewhere 
in the system is, even indirectly, holding on to structures that hold on to 
these structures?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131961427
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131960934
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131945914
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131943632
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131942688
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131937974
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131934285
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
--- End diff --

Yes.  The functionality is incomplete at the moment, but that is because we 
don't have a host that hosts multiple plugins yet.  When you add a parser 
extension and start it ( METRON-942) you are starting a new topology.  

A complete and formal pattern or implementation around Bundle System 
ReInit() would be a next step, but the primitives are there.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131931569
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
--- End diff --

I will Javadoc.  I can imagine a bundle host wanting to reset it's bundle 
environment and reloading.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131831288
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131784556
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/util/BundleProperties.java 
---
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.util;
+
+import org.apache.metron.bundles.bundle.Bundle;
+
+import java.io.*;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.file.Paths;
+import java.util.*;
+
+/**
+ * The BundleProperties class holds all properties which are needed for 
various values to be
+ * available at runtime. It is strongly tied to the startup properties 
needed and is often refer to
+ * as the 'nifi.properties' file. The properties contains keys and values. 
Great care should be
--- End diff --

"is often refer to as the 'nifi.properties' file" -- it is?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131796392
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/bundle/BundleDetails.java 
---
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.bundle;
+
+import org.apache.commons.vfs2.FileObject;
+
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+
+/**
+ * Metadata about a bundle.
+ */
+public class BundleDetails {
+
+private final FileObject bundleFile;
+
+private final BundleCoordinate coordinate;
+private final BundleCoordinate dependencyCoordinate;
+
+private final String buildTag;
+private final String buildRevision;
+private final String buildBranch;
+private final String buildTimestamp;
+private final String buildJdk;
+private final String builtBy;
+
+private BundleDetails(final Builder builder) {
+this.bundleFile = builder.bundleFile;
+this.coordinate = builder.coordinate;
+this.dependencyCoordinate = builder.dependencyCoordinate;
+
+this.buildTag = builder.buildTag;
+this.buildRevision = builder.buildRevision;
+this.buildBranch = builder.buildBranch;
+this.buildTimestamp = builder.buildTimestamp;
+this.buildJdk = builder.buildJdk;
+this.builtBy = builder.builtBy;
+
+if (this.coordinate == null) {
+if (this.bundleFile == null) {
+throw new IllegalStateException("Coordinate cannot be 
null");
+} else {
+throw new IllegalStateException("Coordinate cannot be null 
for " + this.bundleFile.getName());
+}
+}
+
+if (this.bundleFile == null) {
+throw new IllegalStateException("Working directory cannot be 
null for " + this.coordinate.getId());
+}
--- End diff --

Is testing for null sufficient?  Don't we need to make sure the coordinates 
are sensible, the bundlefile exists, perhaps others?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131828675
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinate;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework 
classloaders.
+ */
+public final class BundleClassLoaders {
+
+  private static volatile BundleClassLoaders bundleClassLoaders;
+  private volatile InitContext initContext;
+  private static final Logger logger = 
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+  private final static class InitContext {
+
+private final List extensionDirs;
+private final Map bundles;
+
+private InitContext(
+final List extensionDirs,
+final Map bundles) {
+  this.extensionDirs = extensionDirs;
+  this.bundles = bundles;
+}
+  }
+
+  private BundleClassLoaders() {
+  }
+
+  /**
+   * @return The singleton instance of the BundleClassLoaders
+   */
+  public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+  synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+  bundleClassLoaders = result = new BundleClassLoaders();
+}
+  }
+}
+return result;
+  }
+
+  public static void reset() {
+getInstance().unInit();
+  }
+
+  private void unInit() {
+synchronized (this) {
+  initContext = null;
+}
+  }
+
+  /**
+   * Initializes and loads the BundleClassLoaders. This method must be 
called before the rest of the
+   * methods to access the classloaders are called and it can be safely 
called any number of times
+   * provided the same framework and extension working dirs are used.
+   *
+   * @param fileSystemManager the FileSystemManager
+   * @param extensionsDirs where to find extension artifacts
+   * @param props BundleProperties
+   * @throws FileSystemException if any issue occurs while exploding 
bundle working directories.
+   * @throws java.lang.ClassNotFoundException if unable to load class 
definition
+   * @throws IllegalStateException already initialized with a given pair 
of directories cannot
+   * reinitialize or use a different pair of directories.
+   */
+  public void init(final FileSystemManager fileSystemManager, final 
List extensionsDirs,
+  BundleProperties props)
+  throws FileSystemException, ClassNotFoundException, 
URISyntaxException {
+if (extensionsDirs == null || fileSystemManager == null) {
+  throw new NullPointerException("cannot have empty arguments");
+}
+InitContext ic = initContext;
+if (ic == null) {
+  synchronized (this) {
+ic = initContext;
+if (ic == null) {
+  initContext = ic = load(fileSystemManager, extensionsDirs, 
props);
+}
+  }
+} else {
+  boolean 

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131765763
  
--- Diff: bundles-lib/README.md ---
@@ -0,0 +1,79 @@
+# Apache Metron Bundles
+
+Apache Metron Bundles and this documentation are a derivitave of the 
[Apache Nifi](http://www.nifi.apache.org) 
[NARs](http://nifi.apache.org/developer-guide.html).
+
+When software from many different organizations is all hosted within
+the same environment, Java ClassLoaders quickly
+become a concern. If multiple components have a dependency on the same
+library but each depends on a different
+version, many problems arise, typically resulting in unexpected
+behavior or `NoClassDefFoundError` errors occurring.
+In order to prevent these issues from becoming problematic, Apache NiFi
+introduces the notion of a NiFi Archive, or NAR.  The Apache Metron 
project has adapted and extended the NAR system as
+Apache Metron Bundles.
+
+A BUNDLE allows several components and their dependencies to be packaged
+together into a single package.
+The BUNDLE package is then provided ClassLoader isolation from other 
BUNDLES
+packages. Developers should always deploy their Apache Metron Extensions 
as BUNDLE packages.
+
+To achieve this, a developer creates a new Maven Artifact, which we
+refer to as the BUNDLE artifact. The packaging is
+set to `bundle`. The `dependencies` section of the POM is then created so
+that the BUNDLE has a dependency on all Extension Components that are to 
be included within the BUNDLE.
+
+In order to use a packaging of `bundle`, we must use the 
`bundles-maven-plugin` module.
+This is included by adding the following snippet to the bundle's pom.xml:
+
+
+```xml
+
+
+
+org.apache.metron
+bundles-maven-plugin
+0.4.0
+true
+
+
+
+```
+
+The bundles-maven-plugin is included in the projects created by Apache 
Metron Extension maven archetypes.
+
+
+The BUNDLE is able to have one dependency that is of type `bundle`. If more
+than one dependency is specified that is of type
+`bundle`, then the bundles-maven-plugin will error. If BUNDLE A adds a
+dependency on BUNDLE B, this will *not* result in
+BUNDLE B packaging all of the components of BUNDLE A. Rather, this will add
+a `Bundle-Dependency-Id` element to the `MANIFEST.MF`
+file of BUNDLE A. This will result in setting the ClassLoader of BUNDLE B 
as
+the Parent ClassLoader of BUNDLE A. In this case,
+we refer to BUNDLE B as the _Parent_ of BUNDLE A.
+
+## Per-Instance ClassLoading
+
+The bundles-lib provides the `@RequiresInstanceClassLoading` annotation to 
further expand and isolate the libraries
+available on a component’s classpath. You can annotate a extension class 
with `@RequiresInstanceClassLoading`
+to indicate that the instance ClassLoader for the component requires a 
copy of all the resources in the
+component's BUNDLE ClassLoader. When `@RequiresInstanceClassLoading` is 
not present, the
+instance ClassLoader simply has it's parent ClassLoader set to the BUNDLE 
ClassLoader, rather than
+copying resources.
+
+Because @RequiresInstanceClassLoading copies resources from the BUNDLE 
ClassLoader for each instance of the
+extension, use this capability judiciously in an environment where many 
extensions may be present. If ten instances of one extension are created, all 
classes
+from the component's BUNDLE ClassLoader are loaded into memory ten times. 
This could eventually increase the
+memory footprint significantly when enough instances of the component are 
created.
+
+
+## Apache VFS 
+
+The bundles-lib utilizes the Apache VFS library to for loading bundles.  
Bundles, what are zip files of a known structure, containing 
+jars of dependencies may be loaded by VFS as File Systems *themselves*, 
and in tern each jar in the bundle can be loaded by VFS as a File System.
--- End diff --

editorial nits: "to for loading", "what are zip files", "in tern"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131787199
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/util/FileSystemManagerFactory.java
 ---
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.util;
+
+import org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator;
+import org.apache.commons.vfs2.*;
+import org.apache.commons.vfs2.cache.SoftRefFilesCache;
+import org.apache.commons.vfs2.impl.DefaultFileSystemManager;
+import org.apache.commons.vfs2.impl.FileContentInfoFilenameFactory;
+import org.apache.commons.vfs2.provider.hdfs.HdfsFileProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+
+public class FileSystemManagerFactory {
+  private static final Logger LOG = 
LoggerFactory.getLogger(FileSystemManagerFactory.class);
+
+  /**
+   * Create a FileSystem manager suitable for our purposes.
+   * This manager supports files of the following types:
+   * * res - resource files
+   * * jar
+   * * tar
+   * * bz2
+   * * tgz
+   * * zip
+   * * HDFS
+   * * FTP
+   * * HTTP/S
+   * * file
+   * @return
+   * @throws FileSystemException
+   */
+  public static FileSystemManager createFileSystemManager() throws 
FileSystemException {
+return createFileSystemManager(null);
+  }
+  public static FileSystemManager createFileSystemManager(String 
jarExtensionToRegister) throws FileSystemException {
+DefaultFileSystemManager vfs = new DefaultFileSystemManager();
+
+if(jarExtensionToRegister != null || 
!StringUtils.isBlank(jarExtensionToRegister) ) {
+  vfs.addExtensionMap(jarExtensionToRegister, "jar");
+  vfs.addProvider(jarExtensionToRegister, new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+}
+
+vfs.addProvider("res", new 
org.apache.commons.vfs2.provider.res.ResourceFileProvider());
+vfs.addProvider("zip", new 
org.apache.commons.vfs2.provider.zip.ZipFileProvider());
+vfs.addProvider("gz", new 
org.apache.commons.vfs2.provider.gzip.GzipFileProvider());
+vfs.addProvider("ram", new 
org.apache.commons.vfs2.provider.ram.RamFileProvider());
+vfs.addProvider("file", new 
org.apache.commons.vfs2.provider.local.DefaultLocalFileProvider());
+vfs.addProvider("jar", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("http", new 
org.apache.commons.vfs2.provider.http.HttpFileProvider());
+vfs.addProvider("https", new 
org.apache.commons.vfs2.provider.https.HttpsFileProvider());
+vfs.addProvider("ftp", new 
org.apache.commons.vfs2.provider.ftp.FtpFileProvider());
+vfs.addProvider("ftps", new 
org.apache.commons.vfs2.provider.ftps.FtpsFileProvider());
+vfs.addProvider("war", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("par", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("ear", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("sar", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("ejb3", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("tmp", new 
org.apache.commons.vfs2.provider.temp.TemporaryFileProvider());
+vfs.addProvider("tar", new 
org.apache.commons.vfs2.provider.tar.TarFileProvider());
+vfs.addProvider("tbz2", new 
org.apache.commons.vfs2.provider.tar.TarFileProvider());
+vfs.addProvider("tgz", new 
org.apache.commons.vfs2.provider.tar.TarFileProvider());
+vfs.addProvider("bz2", new 
org.apache.commons.vfs2.provider.bzip2.Bzip2FileProvider());
+vfs.addProvider("hdfs", new HdfsFileProvider());
+vfs.addExtensionMap("jar", "jar");
+vfs.addExtensionMap("zip", "zip");
+vfs.addExtensionMap("gz", "gz");
+

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r131786965
  
--- Diff: 
bundles-lib/src/main/java/org/apache/metron/bundles/util/FileSystemManagerFactory.java
 ---
@@ -0,0 +1,99 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles.util;
+
+import org.apache.accumulo.start.classloader.vfs.UniqueFileReplicator;
+import org.apache.commons.vfs2.*;
+import org.apache.commons.vfs2.cache.SoftRefFilesCache;
+import org.apache.commons.vfs2.impl.DefaultFileSystemManager;
+import org.apache.commons.vfs2.impl.FileContentInfoFilenameFactory;
+import org.apache.commons.vfs2.provider.hdfs.HdfsFileProvider;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+
+public class FileSystemManagerFactory {
+  private static final Logger LOG = 
LoggerFactory.getLogger(FileSystemManagerFactory.class);
+
+  /**
+   * Create a FileSystem manager suitable for our purposes.
+   * This manager supports files of the following types:
+   * * res - resource files
+   * * jar
+   * * tar
+   * * bz2
+   * * tgz
+   * * zip
+   * * HDFS
+   * * FTP
+   * * HTTP/S
+   * * file
+   * @return
+   * @throws FileSystemException
+   */
+  public static FileSystemManager createFileSystemManager() throws 
FileSystemException {
+return createFileSystemManager(null);
+  }
+  public static FileSystemManager createFileSystemManager(String 
jarExtensionToRegister) throws FileSystemException {
+DefaultFileSystemManager vfs = new DefaultFileSystemManager();
+
+if(jarExtensionToRegister != null || 
!StringUtils.isBlank(jarExtensionToRegister) ) {
+  vfs.addExtensionMap(jarExtensionToRegister, "jar");
+  vfs.addProvider(jarExtensionToRegister, new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+}
+
+vfs.addProvider("res", new 
org.apache.commons.vfs2.provider.res.ResourceFileProvider());
+vfs.addProvider("zip", new 
org.apache.commons.vfs2.provider.zip.ZipFileProvider());
+vfs.addProvider("gz", new 
org.apache.commons.vfs2.provider.gzip.GzipFileProvider());
+vfs.addProvider("ram", new 
org.apache.commons.vfs2.provider.ram.RamFileProvider());
+vfs.addProvider("file", new 
org.apache.commons.vfs2.provider.local.DefaultLocalFileProvider());
+vfs.addProvider("jar", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("http", new 
org.apache.commons.vfs2.provider.http.HttpFileProvider());
+vfs.addProvider("https", new 
org.apache.commons.vfs2.provider.https.HttpsFileProvider());
+vfs.addProvider("ftp", new 
org.apache.commons.vfs2.provider.ftp.FtpFileProvider());
+vfs.addProvider("ftps", new 
org.apache.commons.vfs2.provider.ftps.FtpsFileProvider());
+vfs.addProvider("war", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("par", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("ear", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("sar", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("ejb3", new 
org.apache.commons.vfs2.provider.jar.JarFileProvider());
+vfs.addProvider("tmp", new 
org.apache.commons.vfs2.provider.temp.TemporaryFileProvider());
+vfs.addProvider("tar", new 
org.apache.commons.vfs2.provider.tar.TarFileProvider());
+vfs.addProvider("tbz2", new 
org.apache.commons.vfs2.provider.tar.TarFileProvider());
+vfs.addProvider("tgz", new 
org.apache.commons.vfs2.provider.tar.TarFileProvider());
+vfs.addProvider("bz2", new 
org.apache.commons.vfs2.provider.bzip2.Bzip2FileProvider());
+vfs.addProvider("hdfs", new HdfsFileProvider());
--- End diff --

Do we want to remove the Providers above that don't relate to our usages?  
par, ear, sar, etc?


---
If your