[GitHub] incubator-metron issue #542: METRON-873: Stellar string literals do not supp...
Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/542 We need a stellar guide, more suited to learning stellar, than just documenting the facts of it. Something that gets you in the shell and gives you some exercises. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #674: METRON-681 CSVConverter should trim key and values
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/674 Yes. Does X with incoming .. trims all values --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #667: METRON-1061 Add FUZZY_SCORE STELLAR function
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/667#discussion_r131155065
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/TextFunctions.java
---
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import java.util.List;
+import java.util.Locale;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.text.similarity.FuzzyScore;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+public class TextFunctions {
+
+ @Stellar(name = "FUZZY_SCORE",
+ description =
+ "Returns the Fuzzy Score which indicates the similarity score
between two Strings "
+ +
+ "One point is given for every matched character. Subsequent
matches yield two bonus "
+ +
+ "points. A higher score indicates a higher similarity",
+ params = {
+ "string - The full term that should be matched against",
+ "string - The query that will be matched against a term",
+ "string - The IETF BCP 47 language code to use"
+ },
+ returns = "integer representing the score")
+ /**
+ * FuzzyScoreFunction exposes the Apache Commons Text Similarity
FuzzyScore through
+ * Stellar.
+ */
+ public static class FuzzyScoreFunction extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+ if (list.size() < 3) {
+throw new IllegalStateException("FUZZY_SCORE expects three args:
[string, string, string]");
+ }
+ String term = (String) list.get(0);
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #675: METRON-379 STELLAR can differentiate between a value pass...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/675 I'll create a Jira for the NaN thing, unless you have already coded it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #681: METRON-1079 Add NaN as a keyword in STELLAR langua...
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/681 METRON-1079 Add NaN as a keyword in STELLAR language Add "NaN" to STELLAR as a keyword. Add NaN, and tests to verify that it works in line with the java specification. ## Testing Build and Tests should work. Execute statement similar to the added tests in Stellar Shell. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron stellar_nan Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/681.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #681 commit 827566937bfb9527abcd6979d1bb503c78db5f16 Author: Otto Fowler <ottobackwa...@gmail.com> Date: 2017-08-03T16:06:00Z Add NaN as a keyword in STELLAR language --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #682: METRON-1081: Fix Alerts and Ops UI Notices file
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/682 Are the modules/versions pulled in dependent on the version of node or something? My list is different. I have more things, and different versions for some. [licenses.txt](https://github.com/apache/metron/files/1199144/licenses.txt) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 hey @mmiklavc . Looking back at my logs above, no matter that the maven-archetype-plugin was 2.4 in the pom at the time, it shows 3.0.1 in the output. Can you check what version it shows in your output? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 I also may have a fix for the missing catalog file issue ( or a way to run the command differently ) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 thanks @mmiklavc. I don't think I've stated this in the actual PR As a 3rd party ( or whatever you want to call it ) developer, who only makes Parser or possibly other extensions for metron, and does not maintain a fork of metron or change any metron code, I want to manage my extension as a completely self contained package. That means everything that is needed to make my extension work is packaged and installed/deployed as one unit. I do not want my enrichment configurations in some other library, or my indexing configurations for that matter. I may never even have the metron code, only the maven repo. This is a different mindset from the metron was obviously conceived, as a single project. I am not certain what your second config/zookeeper refers to, can you give more detail. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 I DO think that the archetype's input variables could use refactoring. For example, I force the artifact name, and maybe I should not. I do not know why the bundle plugin is looking for the $version, when the pom says $metronVersion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 I hate to ask but are you sure you have the latest code? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 I'm going to try with a new .m2 dir --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 Ok, I tried with a new M2 directory and the can't find local archetype issue returned. maven-archetype-plugin 3.0.1 Which is up from 2.4 resolved that issue, and after that I still cannot reproduce. Is going to 3.0.1 viable? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 ok - so mvn archetype:generate uses the newer version of the plugin if run in an empty directory. If run in a directory where there is a pom with plugin setting for 2.4 and it will honor that. So the issue with the missing catalog file and it not working has to do with the version you are running with and the version you are installing with, and if you are under a pom structure that keeps them the same. I don't know how that helps, but it solves that mystery. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #682: METRON-1081: Fix Alerts and Ops UI Notices file
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/682 I am not sure how to verify or review this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 ```bash ââ[ottofowler@Winterfell] - [~/.m2/repository/org/apache/metron/metron-maven-parser-extension-archetype/0.4.1] - [Thu Aug 03, 18:42] ââ[$]> ll total 80 -rw-r--r-- 1 ottofowler staff 237B Aug 3 18:24 _remote.repositories -rw-r--r-- 1 ottofowler staff29K Aug 3 18:24 metron-maven-parser-extension-archetype-0.4.1.jar -rw-r--r-- 1 ottofowler staff 1.8K Aug 3 18:17 metron-maven-parser-extension-archetype-0.4.1.pom ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 WRT: the archetype directories: I don't see that ( but I'm going to re-build and try again ). I don't use the same options you are using though. Importantly -> the parser version MUST match the METRON version right now. So the 1.0-SNAPSHOT won't build. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 ```bash ââ[ottofowler@Winterfell] - [~/tmp] - [Thu Aug 03, 18:36] ââ[$]> mvn archetype:generate -DarchetypeCatalog=local [INFO] Scanning for projects... [INFO] [INFO] [INFO] Building Maven Stub Project (No POM) 1 [INFO] [INFO] [INFO] >>> maven-archetype-plugin:3.0.1:generate (default-cli) > generate-sources @ standalone-pom >>> [INFO] [INFO] <<< maven-archetype-plugin:3.0.1:generate (default-cli) < generate-sources @ standalone-pom <<< [INFO] [INFO] [INFO] --- maven-archetype-plugin:3.0.1:generate (default-cli) @ standalone-pom --- [INFO] Generating project in Interactive mode [INFO] No archetype defined. Using maven-archetype-quickstart (org.apache.maven.archetypes:maven-archetype-quickstart:1.0) Choose archetype: 1: local -> org.apache.metron:metron-maven-parser-extension-archetype (Apache Maven Parser Extension Archetype for Metron) Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): : 1 Define value for property 'groupId': com.michaelmiklavcic Define value for property 'artifactId': helloparser Define value for property 'version' 1.0-SNAPSHOT: : 0.4.1 Define value for property 'package' com.michaelmiklavcic: : Define value for property 'metronVersion': 0.4.1 Define value for property 'parserClassName' (should match expression '^[A-Z].*$'): Hello Define value for property 'parserName' (should match expression '^[a-z]+[A-Z,a-z]+$'): hello Confirm properties configuration: groupId: com.michaelmiklavcic artifactId: helloparser version: 0.4.1 package: com.michaelmiklavcic metronVersion: 0.4.1 parserClassName: Hello parserName: hello Y: : y [INFO] [INFO] Using following parameters for creating project from Archetype: metron-maven-parser-extension-archetype:0.4.1 [INFO] [INFO] Parameter: groupId, Value: com.michaelmiklavcic [INFO] Parameter: artifactId, Value: helloparser [INFO] Parameter: version, Value: 0.4.1 [INFO] Parameter: package, Value: com.michaelmiklavcic [INFO] Parameter: packageInPathFormat, Value: com/michaelmiklavcic [INFO] Parameter: parserName, Value: hello [INFO] Parameter: package, Value: com.michaelmiklavcic [INFO] Parameter: version, Value: 0.4.1 [INFO] Parameter: metronVersion, Value: 0.4.1 [INFO] Parameter: groupId, Value: com.michaelmiklavcic [INFO] Parameter: parserClassName, Value: Hello [INFO] Parameter: artifactId, Value: helloparser [INFO] Parent element not overwritten in /Users/ottofowler/tmp/helloparser/metron-parser-hello/pom.xml [INFO] Parent element not overwritten in /Users/ottofowler/tmp/helloparser/metron-parser-hello-bundle/pom.xml [INFO] Parent element not overwritten in /Users/ottofowler/tmp/helloparser/metron-parser-hello-assembly/pom.xml [INFO] Project created from Archetype in dir: /Users/ottofowler/tmp/helloparser [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 48.709 s [INFO] Finished at: 2017-08-03T18:37:13-04:00 [INFO] Final Memory: 15M/245M [INFO] ââ[ottofowler@Winterfell] - [~/tmp] - [Thu Aug 03, 18:37] ââ[$]> tree helloparser helloparser âââ metron-parser-hello â  âââ README.md â  âââ pom.xml â  âââ src â  âââ main â  â  âââ config â  â  â  âââ elasticsearch â  â  â  â  âââ hello_index.template â  â  â  âââ zookeeper â  â  â  âââ enrichments â  â  â  â  âââ hello.json â  â  â  âââ indexing â  â  â  â  âââ hello.json â  â  â  âââ parsers â  â  â  âââ hello.json â  â  âââ java â  â  â  âââ com â  â  â  âââ michaelmiklavcic â  â  â  âââ hello â  â  â  âââ HelloParser.java â  â  âââ resou
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 I'm not done the build, so I'll try this again, but this is what I get: ( Note, I don't know how you are getting the packageInPathFormat option ): ââ[ottofowler@Winterfell] - [~/tmp] - [Thu Aug 03, 18:18] ââ[$]> mvn archetype:generate -DarchetypeCatalog=local [INFO] Scanning for projects... [INFO] [INFO] [INFO] Building Maven Stub Project (No POM) 1 [INFO] [INFO] [INFO] >>> maven-archetype-plugin:3.0.1:generate (default-cli) > generate-sources @ standalone-pom >>> [INFO] [INFO] <<< maven-archetype-plugin:3.0.1:generate (default-cli) < generate-sources @ standalone-pom <<< [INFO] [INFO] [INFO] --- maven-archetype-plugin:3.0.1:generate (default-cli) @ standalone-pom --- [INFO] Generating project in Interactive mode [INFO] No archetype defined. Using maven-archetype-quickstart (org.apache.maven.archetypes:maven-archetype-quickstart:1.0) Choose archetype: 1: local -> org.apache.metron:metron-maven-parser-extension-archetype (Apache Maven Parser Extension Archetype for Metron) Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): : 1 Define value for property 'groupId': com.michaelmiklavcic Define value for property 'artifactId': HelloParser Define value for property 'version' 1.0-SNAPSHOT: : 1.0-SNAPSHOT Define value for property 'package' com.michaelmiklavcic: : com.michaelmiklavcic Define value for property 'metronVersion': 0.4.1 Define value for property 'parserClassName' (should match expression '^[A-Z].*$'): HelloParser Define value for property 'parserName' (should match expression '^[a-z]+[A-Z,a-z]+$'): hello Confirm properties configuration: groupId: com.michaelmiklavcic artifactId: HelloParser version: 1.0-SNAPSHOT package: com.michaelmiklavcic metronVersion: 0.4.1 parserClassName: HelloParser parserName: hello Y: : y [INFO] [INFO] Using following parameters for creating project from Archetype: metron-maven-parser-extension-archetype:0.4.1 [INFO] [INFO] Parameter: groupId, Value: com.michaelmiklavcic [INFO] Parameter: artifactId, Value: HelloParser [INFO] Parameter: version, Value: 1.0-SNAPSHOT [INFO] Parameter: package, Value: com.michaelmiklavcic [INFO] Parameter: packageInPathFormat, Value: com/michaelmiklavcic [INFO] Parameter: parserName, Value: hello [INFO] Parameter: package, Value: com.michaelmiklavcic [INFO] Parameter: version, Value: 1.0-SNAPSHOT [INFO] Parameter: metronVersion, Value: 0.4.1 [INFO] Parameter: groupId, Value: com.michaelmiklavcic [INFO] Parameter: parserClassName, Value: HelloParser [INFO] Parameter: artifactId, Value: HelloParser [INFO] Parent element not overwritten in /Users/ottofowler/tmp/HelloParser/metron-parser-hello/pom.xml [INFO] Parent element not overwritten in /Users/ottofowler/tmp/HelloParser/metron-parser-hello-bundle/pom.xml [INFO] Parent element not overwritten in /Users/ottofowler/tmp/HelloParser/metron-parser-hello-assembly/pom.xml [INFO] Project created from Archetype in dir: /Users/ottofowler/tmp/HelloParser [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 03:54 min [INFO] Finished at: 2017-08-03T18:22:44-04:00 [INFO] Final Memory: 16M/309M [INFO] ââ[ottofowler@Winterfell] - [~/tmp] - [Thu Aug 03, 18:22] ââ[$]> ls HelloParser antlr dummy dummy-one dummy-two ââ[ottofowler@Winterfell] - [~/tmp] - [Thu Aug 03, 18:22] ââ[$]> tree HelloParser HelloParser âââ metron-parser-hello â  âââ README.md â  âââ pom.xml â  âââ src â  âââ main â  â  âââ config â  â  â  âââ elasticsearch â  â  â  â  âââ hello_index.template â  â  â  âââ zookeeper â  â  â  âââ enrichments â  â  â  â  âââ hello.json â  â  â  âââ indexing â  â  â  â  âââ hello.json â  â  â  âââ parsers â  â
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 Ok - I think you are running find from /usr/metron/$V/ So - to explain the other config/zookeeper So let's pretend there are two things here: 1. metron-parsers 2. parser-extensions metron-parsers ( as described way above in PR descriptions) is: - The Uber Jar loaded by storm - Has all the Parser base classes and Interfaces - Has the 'Raw' Parsers -> CSV, JSONMap, GROK Any configuration for those parsers are installed into the root /usr/metron/$V/config... path. They always where actually. I did not move them, because these are not extensions. Now, if you want to say that there should not be a default configuration for JSONMap under parsers... OK, but we already HAD these before. https://user-images.githubusercontent.com/551/28945840-292df89c-7876-11e7-8c4d-5bbbdaf51ea4.png;> --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 I left the Parser off of the HelloParser input, sorry --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 ```bash ââ[ottofowler@Winterfell] - [~/tmp/HelloParser] - [Thu Aug 03, 19:12] ââ[$]> cat ~/.m2/repository/archetype-catalog.xml http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0 http://maven.apache.org/xsd/archetype-catalog-1.0.0.xsd; xmlns="http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;> org.apache.metron metron-maven-parser-extension-archetype 0.4.1 Apache Maven Parser Extension Archetype for Metron ``` ```bash ââ[ottofowler@Winterfell] - [~/tmp/HelloParser] - [Thu Aug 03, 19:13] ââ[$]> cat ~/.m2/archetype-catalog.xml http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0 http://maven.apache.org/xsd/archetype-catalog-1.0.0.xsd; xmlns="http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;> org.apache.nifi nifi-service-bundle-archetype 1.2.0-SNAPSHOT Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. org.apache.nifi nifi-processor-bundle-archetype 1.2.0-SNAPSHOT Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. org.apache.metron metron-maven-parser-extension-archetype 0.4.1 Apache Maven Parser Extension Archetype for Metron ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #679: METRON-1075: Add faceted search capabilities
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/679 +1 works as described. Great Job! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #667: METRON-1061 Add FUZZY_SCORE STELLAR function
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/667#discussion_r131142516
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/TextFunctions.java
---
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import java.util.List;
+import java.util.Locale;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.text.similarity.FuzzyScore;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+public class TextFunctions {
+
+ @Stellar(name = "FUZZY_SCORE",
+ description =
+ "Returns the Fuzzy Score which indicates the similarity score
between two Strings "
+ +
+ "One point is given for every matched character. Subsequent
matches yield two bonus "
+ +
+ "points. A higher score indicates a higher similarity",
+ params = {
+ "string - The full term that should be matched against",
+ "string - The query that will be matched against a term",
+ "string - The IETF BCP 47 language code to use"
+ },
+ returns = "integer representing the score")
+ /**
+ * FuzzyScoreFunction exposes the Apache Commons Text Similarity
FuzzyScore through
+ * Stellar.
+ */
+ public static class FuzzyScoreFunction extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+ if (list.size() < 3) {
+throw new IllegalStateException("FUZZY_SCORE expects three args:
[string, string, string]");
+ }
+ String term = (String) list.get(0);
--- End diff --
1. The (CAST)list.get(0) is the common pattern in our stellar code. I
believe I have asked before in another pr maybe why we don't use the conversion
utils.
2. I think we want to return 0 for invalid args.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #667: METRON-1061 Add FUZZY_SCORE STELLAR function
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/667#discussion_r131143225
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/TextFunctions.java
---
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import java.util.List;
+import java.util.Locale;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.text.similarity.FuzzyScore;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+public class TextFunctions {
+
+ @Stellar(name = "FUZZY_SCORE",
+ description =
+ "Returns the Fuzzy Score which indicates the similarity score
between two Strings "
+ +
+ "One point is given for every matched character. Subsequent
matches yield two bonus "
+ +
+ "points. A higher score indicates a higher similarity",
+ params = {
+ "string - The full term that should be matched against",
+ "string - The query that will be matched against a term",
+ "string - The IETF BCP 47 language code to use"
+ },
+ returns = "integer representing the score")
+ /**
+ * FuzzyScoreFunction exposes the Apache Commons Text Similarity
FuzzyScore through
+ * Stellar.
+ */
+ public static class FuzzyScoreFunction extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+ if (list.size() < 3) {
+throw new IllegalStateException("FUZZY_SCORE expects three args:
[string, string, string]");
+ }
+ String term = (String) list.get(0);
--- End diff --
We really need a pattern or a helper class for stellar for variables, that
everyone uses.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #667: METRON-1061 Add FUZZY_SCORE STELLAR function
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/667#discussion_r131146245
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/TextFunctions.java
---
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import java.util.List;
+import java.util.Locale;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.text.similarity.FuzzyScore;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+public class TextFunctions {
+
+ @Stellar(name = "FUZZY_SCORE",
+ description =
+ "Returns the Fuzzy Score which indicates the similarity score
between two Strings "
+ +
+ "One point is given for every matched character. Subsequent
matches yield two bonus "
+ +
+ "points. A higher score indicates a higher similarity",
+ params = {
+ "string - The full term that should be matched against",
+ "string - The query that will be matched against a term",
+ "string - The IETF BCP 47 language code to use"
+ },
+ returns = "integer representing the score")
+ /**
+ * FuzzyScoreFunction exposes the Apache Commons Text Similarity
FuzzyScore through
+ * Stellar.
+ */
+ public static class FuzzyScoreFunction extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+ if (list.size() < 3) {
+throw new IllegalStateException("FUZZY_SCORE expects three args:
[string, string, string]");
+ }
+ String term = (String) list.get(0);
--- End diff --
Also, the return should consider the common usage, I imagine something like:
IF (FUZZY_SCORE(fld,qry,'EN') > 4) THEN SET SOME FIELD
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #686: METRON-711 STELLAR SHELL Do not set variables when except...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/686 @cestella do you have any inputs to the questions posed in the pr description? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #681: METRON-1079 Add NaN as a keyword in STELLAR language
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/681 We are not 'doing' the math here. What we are doing is introducing the ability to use NaN in an expression correctly. Java is doing the math. The tests are to verify that through stellar, the specified behavior still is adhered to. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #667: METRON-1061 Add FUZZY_SCORE STELLAR function
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/667#discussion_r133247450 --- Diff: metron-stellar/stellar-common/README.md --- @@ -411,6 +412,14 @@ In the core language functions, we support basic functional programming primitiv * format - string * arguments... - object(s) * Returns: A formatted string. + +### `FUZZY_SCORE` + * Description: Returns the Fuzzy Score which indicates the similarity score between two strings. One point is given for every matched character. Subsequent matches yield two bonus points. A higher score indicates a higher similarity. + * Input: +* string - The full term that should be matched against. +* string - The query that will be matched against a term. +* string - The IETF BCP 47 language code to use. --- End diff -- I did both --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #667: METRON-1061 Add FUZZY_SCORE STELLAR function
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/667#discussion_r133273527
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/TextFunctions.java
---
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.text.similarity.FuzzyScore;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+public class TextFunctions {
+
+ @Stellar(name = "GET_AVAILABLE_LANGUAGE_TAGS",
--- End diff --
I always forget about the undocumented Namespace feature... I sure will do
that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 *[bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java, line 43 at r2](https://reviewable.io:443/reviews/apache/metron/530#-Kr7wzDH9halWFNY1USd-r2-43:-KrRDdWkwG4RlkudNqzn:b-7wc8nh) ([raw file](https://github.com/apache/metron/blob/571168d897cdd10bc616cef92b613763f76a1842/bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java#L43)):* Previously, ottobackwards wrote⦠I don't like them either. I thought about DI and other things, but I was avoiding major changes of that type. I'm more willing now to part ways. I have refactored the ExtensionManager, I'm going to refactor this as well to make it all simpler. --- *Comments from [Reviewable](https://reviewable.io:443/reviews/apache/metron/530)* --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #690: METRON-1091 Package STELLAR shell as stand alone
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/690 @JonZeolla Thanks for the review! There are several questions in the description that I think need to fall out of the review, one of them is where or how to document this. Any ideas? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 @mmiklavc keep in mind that the proper scope for the document is building new parsers that are 'part' of metron for METRON-777. METRON-942 has the capability to actually install and deploy archetype products into the system post install. I don't want to conflate the two in this PR, since I had to separate them in the first place. Do you have a timeframe for when I can see what you have in mind? I'm looking forward to your help --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 We discussed this higher up in the pr and decided not to --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #439: METRON-571 add stellar external functions feature - code ...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/439 HI do we have an update on the status of this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133061877
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,436 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * Scans through the classpath to load all extension components using the
service provider API and
+ * running through all classloaders (root, BUNDLEs).
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+ private static final Logger logger =
LoggerFactory.getLogger(ExtensionManager.class);
+
+ public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new
BundleCoordinates(
+ BundleCoordinates.DEFAULT_GROUP, "system",
BundleCoordinates.DEFAULT_VERSION);
+
+ // Maps a service definition (interface) to those classes that implement
the interface
+ private static final Map<Class, Set> definitionMap = new
HashMap<>();
+
+ private static final Map<String, List> classNameBundleLookup =
new HashMap<>();
+ private static final Map<BundleCoordinates, Bundle>
bundleCoordinateBundleLookup = new HashMap<>();
+ private static final Map<ClassLoader, Bundle> classLoaderBundleLookup =
new HashMap<>();
+
+ private static final Set requiresInstanceClassLoading = new
HashSet<>();
+ private static final Map<String, ClassLoader> instanceClassloaderLookup
= new ConcurrentHashMap<>();
+
+ private static AtomicBoolean inited = new AtomicBoolean(false);
+
+ // should initialize class definitions
+ public static void initClassDefinitions(final List classes) {
+if (classes != null) {
+ for (Class clazz : classes) {
+definitionMap.put(clazz, new HashSet<>());
+ }
+}
+inited.set(true);
+ }
+
+ public static void resetClassDefinitions() {
+definitionMap.clear();
+inited.set(false);
+ }
+
+ /**
+ * Loads all extension class types that can be found on the bootstrap
classloader and by creating
+ * classloaders for all BUNDLES found within the classpath.
+ *
+ * @param bundles the bundles to scan through in search of extensions
+ */
+ public static void discoverExtensions(final Bundle systemBundle, final
Set bundles)
+ throws NotInitializedException {
+checkInitialized();
+// get the current context class loader
+ClassLoader currentContextClassLoader =
Thread.currentThread().getContextClassLoader();
+
+// load the system bundle first so that any extensions found in JARs
directly in lib will be registered as
+// being from the system bundle and not from all the other Bundles
+loadExtensions(systemBun
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133050916
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+
+import java.util.*;
+import java.util.function.BiFunction;
+
+public class ExtensionMapping {
+
+ private final Map<String, Map<String, Set>>
extensionNameMap = new HashMap<>();
--- End diff --
Which I have now done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #439: METRON-571 add stellar external functions feature - code ...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/439 As pointed out these should be in the management package --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 @mattf-horton I really like how this is evolving. One thing I have been thinking of since adding the BundleSystem interface ( which should be the main external interface ) is that I would like to refactor the packages, and introduce a .core, with the current roots and subfolders, exposing only the BundleSystem at the root. Do you have any thoughts on this? I know it assumes you have caught up with and are OK with the BundleSystem sorry --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133033031
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+
+import java.util.*;
+import java.util.function.BiFunction;
+
+public class ExtensionMapping {
+
+ private final Map<String, Map<String, Set>>
extensionNameMap = new HashMap<>();
--- End diff --
Note - that with the BundleSystem, the properties file IS now used for
loading, unlike before - IF the classes are not explicitly set.
I will have to write something to explain the ExtensionMappings vs. the
ExtensionManager
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133083369
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/BundleSystem.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+import java.lang.invoke.MethodHandles;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.FileSystemManagerFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * High level interface to the Bundle System. While you may want to use
the lower level classes it
+ * is not required, as BundleSystem provides the base required interface
for initializing the system
+ * and instantiating classes
+ */
+public class BundleSystem {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Builder for a BundleSystem. only {@link BundleProperties} are
required. Beyond that, the
+ * BundleProperties, if they are the only parameter must have archive
extension and bundle
+ * extension types properties present.
+ */
+ public static class Builder {
+
+private BundleProperties properties;
+private FileSystemManager fileSystemManager;
+private List extensionClasses = new LinkedList<>();
+private Bundle systemBundle;
+
+/**
+ * The BundleProperties to use. Unless other builder parameters
override options
+ * (withExtensionClasses ), they must have archive extension and
bundle extensions types
+ * specified
+ *
+ * @param properties The BundleProperties
+ * @return Builder
+ */
+public Builder withBundleProperties(BundleProperties properties) {
+ this.properties = properties;
+ return this;
+}
+
+/**
+ * Provide a {@link FileSystemManager} to overide the default
+ *
+ * @param fileSystemManager override
+ * @return Builder
+ */
+public Builder withFileSystemManager(FileSystemManager
fileSystemManager) {
+ this.fileSystemManager = fileSystemManager;
+ return this;
+}
+
+/**
+ * Provide Extension Classes. If not provided with this override then
the classes will be
+ * configured from the BundleProperties. If provided, the properties
file will not be used.
+ *
+ * @param extensionClasses override
+ * @return Builder
+ */
+public Builder withExtensionClasses(List extensionClasses) {
+ this.extensionClasses.addAll(extensionClasses);
+ return this;
+}
+
+/**
+ * Provide a SystemBundle. If not provided with this override then
the default SystemBundle
+ * will be created.
+ */
+public Builder withSystemBundle(Bundle systemBundle) {
+ this.systemBundle = systemBundle;
+ return this;
+}
+
+/**
+ * Builds a new BundleSystem.
+ *
+ * @return BundleSystem
+ * @throws NotInitializedException if any errors happen during build
+ */
+public BundleSystem build() throws NotInitializedException {
+ if (this.properties == null) {
+throw new IllegalArgumentException("BundleProperties are
required");
--- End diff --
I will make it more clear they will not be used for the classes, but will
be used for other things
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enab
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133077758
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java
---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * THREAD SAFE
+ */
+public class BundleThreadContextClassLoader extends URLClassLoader {
+static final Logger LOG =
LoggerFactory.getLogger(BundleThreadContextClassLoader.class);
+static final ContextSecurityManager contextSecurityManager = new
ContextSecurityManager();
+private final ClassLoader forward = ClassLoader.getSystemClassLoader();
+
+private BundleThreadContextClassLoader() {
+super(new URL[0]);
+}
+
+@Override
+public void clearAssertionStatus() {
+lookupClassLoader().clearAssertionStatus();
+}
+
+@Override
+public URL getResource(String name) {
+return lookupClassLoader().getResource(name);
+}
+
+@Override
+public InputStream getResourceAsStream(String name) {
+return lookupClassLoader().getResourceAsStream(name);
+}
+
+@Override
+public Enumeration getResources(String name) throws IOException {
+return lookupClassLoader().getResources(name);
+}
+
+@Override
+public Class loadClass(String name) throws ClassNotFoundException {
+return lookupClassLoader().loadClass(name);
+}
+
+@Override
+public void setClassAssertionStatus(String className, boolean enabled)
{
+lookupClassLoader().setClassAssertionStatus(className, enabled);
+}
+
+@Override
+public void setDefaultAssertionStatus(boolean enabled) {
+lookupClassLoader().setDefaultAssertionStatus(enabled);
+}
+
+@Override
+public void setPackageAssertionStatus(String packageName, boolean
enabled) {
+lookupClassLoader().setPackageAssertionStatus(packageName,
enabled);
+}
+
+private ClassLoader lookupClassLoader() {
+final Class[] classStack =
contextSecurityManager.getExecutionStack();
+
+for (Class currentClass : classStack) {
+final Class bundleClass = findBundleClass(currentClass);
+if (bundleClass != null) {
+final ClassLoader desiredClassLoader =
bundleClass.getClassLoader();
+
+// When new Threads are created, the new Thread inherits
the ClassLoaderContext of
+// the caller. However, the call stack of that new Thread
may not trace back to any app-specific
+// code. Therefore, the BundleThreadContextClassLoader
will be unable to find the appropriate Bundle
+// ClassLoader. As a result, we want to set the
ContextClassLoader to the Bundle ClassLoader that
+// contains the class or resource that we are looking for.
+// This locks the current Thread into the appropriate
Bundle ClassLoader Context. The framework will change
+// the ContextClassLoader back to the
BundleThreadContextClassLoader as appropriate via the
+//
+// TL;DR
+// We need to make sure the classloader for the thread is
se
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133078904
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/BundleSystem.java ---
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+import java.lang.invoke.MethodHandles;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.LinkedList;
+import java.util.List;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.FileSystemManagerFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * High level interface to the Bundle System. While you may want to use
the lower level classes it
+ * is not required, as BundleSystem provides the base required interface
for initializing the system
+ * and instantiating classes
+ */
+public class BundleSystem {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Builder for a BundleSystem. only {@link BundleProperties} are
required. Beyond that, the
+ * BundleProperties, if they are the only parameter must have archive
extension and bundle
+ * extension types properties present.
+ */
+ public static class Builder {
+
+private BundleProperties properties;
+private FileSystemManager fileSystemManager;
+private List extensionClasses = new LinkedList<>();
+private Bundle systemBundle;
+
+/**
+ * The BundleProperties to use. Unless other builder parameters
override options
+ * (withExtensionClasses ), they must have archive extension and
bundle extensions types
+ * specified
+ *
+ * @param properties The BundleProperties
+ * @return Builder
+ */
+public Builder withBundleProperties(BundleProperties properties) {
+ this.properties = properties;
+ return this;
+}
+
+/**
+ * Provide a {@link FileSystemManager} to overide the default
+ *
+ * @param fileSystemManager override
+ * @return Builder
+ */
+public Builder withFileSystemManager(FileSystemManager
fileSystemManager) {
+ this.fileSystemManager = fileSystemManager;
+ return this;
+}
+
+/**
+ * Provide Extension Classes. If not provided with this override then
the classes will be
+ * configured from the BundleProperties. If provided, the properties
file will not be used.
+ *
+ * @param extensionClasses override
+ * @return Builder
+ */
+public Builder withExtensionClasses(List extensionClasses) {
+ this.extensionClasses.addAll(extensionClasses);
+ return this;
+}
+
+/**
+ * Provide a SystemBundle. If not provided with this override then
the default SystemBundle
+ * will be created.
+ */
+public Builder withSystemBundle(Bundle systemBundle) {
+ this.systemBundle = systemBundle;
+ return this;
+}
+
+/**
+ * Builds a new BundleSystem.
+ *
+ * @return BundleSystem
+ * @throws NotInitializedException if any errors happen during build
+ */
+public BundleSystem build() throws NotInitializedException {
+ if (this.properties == null) {
+throw new IllegalArgumentException("BundleProperties are
required");
--- End diff --
We need the properties for other things
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not wo
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133079542
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+ private static volatile ExtensionManager extensionManager;
+ private volatile InitContext initContext;
+
+ private static final Logger logger =
LoggerFactory.getLogger(ExtensionManager.class);
+
+ public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new
BundleCoordinates(
+ BundleCoordinates.DEFAULT_GROUP, "system",
BundleCoordinates.DEFAULT_VERSION);
+
+ private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that
implement the interface
+private final Map<Class, Set> definitionMap;
+private final Map<String, List> classNameBundleLookup;
+private final Map<BundleCoordinates, Bundle>
bundleCoordinateBundleLookup;
+private final Map<ClassLoader, Bundle> classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map<String, ClassLoader> instanceClassloaderLookup;
+
+private InitContext(Map<Class, Set> definitionMap,
+Map<String, List> classNameBundleLookup,
+Map<BundleCoordinates, Bundle> bundleCoordinateBundleLookup,
+Map<ClassLoader, Bundle> classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map<String, ClassLoader> instanceClassloaderLookup) {
+
+ this.definitionMap =
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+ this.classNameBundleLookup = ImmutableCollectionUtils
+ .immutableMapOfLists(classNameBundleLookup);
+ this.bundleCoordinateBundleLookup =
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+ this.classLoaderBundleLookup =
ImmutableMap.copyOf(classLoaderBundleLookup);
+ this.requiresInstanceClassLoading =
ImmutableSet.copyOf(requiresInstanceClassLoading);
+ this.instanceClassloaderLo
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133079502
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+ private static volatile ExtensionManager extensionManager;
+ private volatile InitContext initContext;
+
+ private static final Logger logger =
LoggerFactory.getLogger(ExtensionManager.class);
+
+ public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new
BundleCoordinates(
+ BundleCoordinates.DEFAULT_GROUP, "system",
BundleCoordinates.DEFAULT_VERSION);
+
+ private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that
implement the interface
+private final Map<Class, Set> definitionMap;
+private final Map<String, List> classNameBundleLookup;
+private final Map<BundleCoordinates, Bundle>
bundleCoordinateBundleLookup;
+private final Map<ClassLoader, Bundle> classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map<String, ClassLoader> instanceClassloaderLookup;
+
+private InitContext(Map<Class, Set> definitionMap,
+Map<String, List> classNameBundleLookup,
+Map<BundleCoordinates, Bundle> bundleCoordinateBundleLookup,
+Map<ClassLoader, Bundle> classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map<String, ClassLoader> instanceClassloaderLookup) {
+
+ this.definitionMap =
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+ this.classNameBundleLookup = ImmutableCollectionUtils
+ .immutableMapOfLists(classNameBundleLookup);
+ this.bundleCoordinateBundleLookup =
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+ this.classLoaderBundleLookup =
ImmutableMap.copyOf(classLoaderBundleLookup);
+ this.requiresInstanceClassLoading =
ImmutableSet.copyOf(requiresInstanceClassLoading);
+ this.instanceClassloaderLo
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133079554
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+ private static volatile ExtensionManager extensionManager;
+ private volatile InitContext initContext;
+
+ private static final Logger logger =
LoggerFactory.getLogger(ExtensionManager.class);
+
+ public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new
BundleCoordinates(
+ BundleCoordinates.DEFAULT_GROUP, "system",
BundleCoordinates.DEFAULT_VERSION);
+
+ private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that
implement the interface
+private final Map<Class, Set> definitionMap;
+private final Map<String, List> classNameBundleLookup;
+private final Map<BundleCoordinates, Bundle>
bundleCoordinateBundleLookup;
+private final Map<ClassLoader, Bundle> classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map<String, ClassLoader> instanceClassloaderLookup;
+
+private InitContext(Map<Class, Set> definitionMap,
+Map<String, List> classNameBundleLookup,
+Map<BundleCoordinates, Bundle> bundleCoordinateBundleLookup,
+Map<ClassLoader, Bundle> classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map<String, ClassLoader> instanceClassloaderLookup) {
+
+ this.definitionMap =
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+ this.classNameBundleLookup = ImmutableCollectionUtils
+ .immutableMapOfLists(classNameBundleLookup);
+ this.bundleCoordinateBundleLookup =
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+ this.classLoaderBundleLookup =
ImmutableMap.copyOf(classLoaderBundleLookup);
+ this.requiresInstanceClassLoading =
ImmutableSet.copyOf(requiresInstanceClassLoading);
+ this.instanceClassloaderLo
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133079476
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+ private static volatile ExtensionManager extensionManager;
+ private volatile InitContext initContext;
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133079524
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+ private static volatile ExtensionManager extensionManager;
+ private volatile InitContext initContext;
+
+ private static final Logger logger =
LoggerFactory.getLogger(ExtensionManager.class);
+
+ public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new
BundleCoordinates(
+ BundleCoordinates.DEFAULT_GROUP, "system",
BundleCoordinates.DEFAULT_VERSION);
+
+ private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that
implement the interface
+private final Map<Class, Set> definitionMap;
+private final Map<String, List> classNameBundleLookup;
+private final Map<BundleCoordinates, Bundle>
bundleCoordinateBundleLookup;
+private final Map<ClassLoader, Bundle> classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map<String, ClassLoader> instanceClassloaderLookup;
+
+private InitContext(Map<Class, Set> definitionMap,
+Map<String, List> classNameBundleLookup,
+Map<BundleCoordinates, Bundle> bundleCoordinateBundleLookup,
+Map<ClassLoader, Bundle> classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map<String, ClassLoader> instanceClassloaderLookup) {
+
+ this.definitionMap =
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+ this.classNameBundleLookup = ImmutableCollectionUtils
+ .immutableMapOfLists(classNameBundleLookup);
+ this.bundleCoordinateBundleLookup =
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+ this.classLoaderBundleLookup =
ImmutableMap.copyOf(classLoaderBundleLookup);
+ this.requiresInstanceClassLoading =
ImmutableSet.copyOf(requiresInstanceClassLoading);
+ this.instanceClassloaderLo
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079584 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java --- @@ -0,0 +1,152 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import org.apache.metron.bundles.bundle.BundleCoordinates; + +import java.util.*; +import java.util.function.BiFunction; + +/** + * The ExtensionMapping represents a mapping of the extensions available to the system. + * It is the product of the BundleMapper. + * + * It is NOT used at runtime for loading extensions, rather it may be used by a system to --- End diff -- done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133079513
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+ private static volatile ExtensionManager extensionManager;
+ private volatile InitContext initContext;
+
+ private static final Logger logger =
LoggerFactory.getLogger(ExtensionManager.class);
+
+ public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new
BundleCoordinates(
+ BundleCoordinates.DEFAULT_GROUP, "system",
BundleCoordinates.DEFAULT_VERSION);
+
+ private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that
implement the interface
+private final Map<Class, Set> definitionMap;
+private final Map<String, List> classNameBundleLookup;
+private final Map<BundleCoordinates, Bundle>
bundleCoordinateBundleLookup;
+private final Map<ClassLoader, Bundle> classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map<String, ClassLoader> instanceClassloaderLookup;
+
+private InitContext(Map<Class, Set> definitionMap,
+Map<String, List> classNameBundleLookup,
+Map<BundleCoordinates, Bundle> bundleCoordinateBundleLookup,
+Map<ClassLoader, Bundle> classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map<String, ClassLoader> instanceClassloaderLookup) {
+
+ this.definitionMap =
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+ this.classNameBundleLookup = ImmutableCollectionUtils
+ .immutableMapOfLists(classNameBundleLookup);
+ this.bundleCoordinateBundleLookup =
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+ this.classLoaderBundleLookup =
ImmutableMap.copyOf(classLoaderBundleLookup);
+ this.requiresInstanceClassLoading =
ImmutableSet.copyOf(requiresInstanceClassLoading);
+ this.instanceClassloaderLo
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079462 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,540 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.metron.bundles; + + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.DummyFileObject; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.ImmutableCollectionUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A Singleton class for scanning through the classpath to load all extension components using + * the ClassIndex and running through all classloaders (root, BUNDLEs). + * + * + * @ThreadSafe - is immutable --- End diff -- done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133076687
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java
---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.InvocationTargetException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * THREAD SAFE
+ */
+public class BundleThreadContextClassLoader extends URLClassLoader {
+static final Logger LOG =
LoggerFactory.getLogger(BundleThreadContextClassLoader.class);
+static final ContextSecurityManager contextSecurityManager = new
ContextSecurityManager();
+private final ClassLoader forward = ClassLoader.getSystemClassLoader();
+
--- End diff --
If we can't get rid of all the static init, at least we can get rid of as
much as possible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133078228
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,540 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.DummyFileObject;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.ImmutableCollectionUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A Singleton class for scanning through the classpath to load all
extension components using
+ * the ClassIndex and running through all classloaders (root, BUNDLEs).
+ *
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+ private static volatile ExtensionManager extensionManager;
+ private volatile InitContext initContext;
+
+ private static final Logger logger =
LoggerFactory.getLogger(ExtensionManager.class);
+
+ public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new
BundleCoordinates(
+ BundleCoordinates.DEFAULT_GROUP, "system",
BundleCoordinates.DEFAULT_VERSION);
+
+ private static final class InitContext {
+
+// Maps a service definition (interface) to those classes that
implement the interface
+private final Map<Class, Set> definitionMap;
+private final Map<String, List> classNameBundleLookup;
+private final Map<BundleCoordinates, Bundle>
bundleCoordinateBundleLookup;
+private final Map<ClassLoader, Bundle> classLoaderBundleLookup;
+private final Set requiresInstanceClassLoading;
+private final Map<String, ClassLoader> instanceClassloaderLookup;
+
+private InitContext(Map<Class, Set> definitionMap,
+Map<String, List> classNameBundleLookup,
+Map<BundleCoordinates, Bundle> bundleCoordinateBundleLookup,
+Map<ClassLoader, Bundle> classLoaderBundleLookup,
+Set requiresInstanceClassLoading,
+Map<String, ClassLoader> instanceClassloaderLookup) {
+
+ this.definitionMap =
ImmutableCollectionUtils.immutableMapOfSets(definitionMap);
+ this.classNameBundleLookup = ImmutableCollectionUtils
+ .immutableMapOfLists(classNameBundleLookup);
+ this.bundleCoordinateBundleLookup =
ImmutableMap.copyOf(bundleCoordinateBundleLookup);
+ this.classLoaderBundleLookup =
ImmutableMap.copyOf(classLoaderBundleLookup);
+ this.requiresInstanceClassLoading =
ImmutableSet.copyOf(requiresInstanceClassLoading);
+ this.instanceClassloaderLooku
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r133079379 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java --- @@ -0,0 +1,213 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + +import java.io.IOException; +import java.io.InputStream; +import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.Enumeration; +import java.util.List; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.util.BundleProperties; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * THREAD SAFE --- End diff -- gone --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133079324
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,376 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework
classloaders.
+ */
+public final class BundleClassLoaders {
+
+ private static volatile BundleClassLoaders bundleClassLoaders;
+ private volatile InitContext initContext;
+ private static final Logger logger =
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+ /**
+ * Holds the context from {@code BundleClassLoaders} initialization,
+ * being the coordinate to bundle mapping.
+ *
+ * After initialization these are not changed, and as such they
+ * are immutable.
+ *
+ */
+ private final static class InitContext {
+
+private final List extensionDirs;
+private final Map<String, Bundle> bundles;
+private final BundleProperties properties;
+
+private InitContext(
+final List extensionDirs,
+final Map<String, Bundle> bundles,
+final BundleProperties properties) {
+ this.extensionDirs = ImmutableList.copyOf(extensionDirs);
+ this.bundles = ImmutableMap.copyOf(bundles);
+ this.properties = properties;
+}
+ }
+
+ private BundleClassLoaders() {
+ }
+
+ /**
+ * @return The singleton instance of the BundleClassLoaders
+ */
+ public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+ synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+ bundleClassLoaders = new BundleClassLoaders();
+ result = bundleClassLoaders;
+}
+ }
+}
+return result;
+ }
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133079362
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -0,0 +1,376 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import java.net.URISyntaxException;
+import java.util.*;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.BundleSelector;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.BundleUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A singleton class used to initialize the extension and framework
classloaders.
+ */
+public final class BundleClassLoaders {
+
+ private static volatile BundleClassLoaders bundleClassLoaders;
+ private volatile InitContext initContext;
+ private static final Logger logger =
LoggerFactory.getLogger(BundleClassLoaders.class);
+
+ /**
+ * Holds the context from {@code BundleClassLoaders} initialization,
+ * being the coordinate to bundle mapping.
+ *
+ * After initialization these are not changed, and as such they
+ * are immutable.
+ *
+ */
+ private final static class InitContext {
+
+private final List extensionDirs;
+private final Map<String, Bundle> bundles;
+private final BundleProperties properties;
+
+private InitContext(
+final List extensionDirs,
+final Map<String, Bundle> bundles,
+final BundleProperties properties) {
+ this.extensionDirs = ImmutableList.copyOf(extensionDirs);
+ this.bundles = ImmutableMap.copyOf(bundles);
+ this.properties = properties;
+}
+ }
+
+ private BundleClassLoaders() {
+ }
+
+ /**
+ * @return The singleton instance of the BundleClassLoaders
+ */
+ public static BundleClassLoaders getInstance() {
+BundleClassLoaders result = bundleClassLoaders;
+if (result == null) {
+ synchronized (BundleClassLoaders.class) {
+result = bundleClassLoaders;
+if (result == null) {
+ bundleClassLoaders = new BundleClassLoaders();
+ result = bundleClassLoaders;
+}
+ }
+}
+return result;
+ }
+
+ /**
+ * Uninitializes the BundleClassloaders. After calling this
init must be called
+ * before the rest of the methods are called afterwards.
+ * This is for TESTING ONLY at this time. Reset does not unload or
clear any loaded classloaders.
+ */
+ public static void reset() {
+synchronized (BundleClassLoaders.class) {
+ getInstance().unInit();
+ bundleClassLoaders = null;
+}
+ }
+
+ private void unInit() {
+synchronized (this) {
+ if(initContext != null) {
+initContext = null;
+ }
+}
+ }
+
+ /**
+ * Initializes and loads the BundleClassLoaders. This method must be
called before the rest of the
+ * methods to access the classloaders are called. Multiple calls to
this method will have no effect,
+ * unless a different set of extension directories is passed, which will
result in an IllegaStateException
+ *
+ * @param fileSystemManager the FileSystemManager
+ * @param extensionsDirs w
[GitHub] metron issue #681: METRON-1079 Add NaN as a keyword in STELLAR language
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/681 @cestella and I were talking about null variables and how NaN was being used, and that we probably needed NaN as a keyword. So I took a stab at it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #667: METRON-1061 Add FUZZY_SCORE STELLAR function
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/667#discussion_r133117091 --- Diff: metron-stellar/stellar-common/README.md --- @@ -411,6 +412,14 @@ In the core language functions, we support basic functional programming primitiv * format - string * arguments... - object(s) * Returns: A formatted string. + +### `FUZZY_SCORE` + * Description: Returns the Fuzzy Score which indicates the similarity score between two strings. One point is given for every matched character. Subsequent matches yield two bonus points. A higher score indicates a higher similarity. + * Input: +* string - The full term that should be matched against. +* string - The query that will be matched against a term. +* string - The IETF BCP 47 language code to use. --- End diff -- you mean to the doc? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #689: METRON-1102: Add support for ingesting cybox URI observab...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/689 As for where... over in METRON-777 @mmiklavc is talking about a better documentation area... this may fall into that --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #689: METRON-1102: Add support for ingesting cybox URI observab...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/689 > Can we handle just Cybox What I mean is, can we import a file that is **ONLY** cybox observables, without Stix. Stix supports cybox. But they are separate things. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #689: METRON-1102: Add support for ingesting cybox URI observab...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/689 I would like to see at least the version of Stix and Cybox that is supported documented. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #689: METRON-1102: Add support for ingesting cybox URI observab...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/689 +1 by inspection. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 The management ui works with zookeeper. You will be able to work with it. Adding support for installing extensions through the UI is a follow on, but I am not sure that I can do it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 in fact, you will see more in the management ui, since the extension parsers all have default configurations and will be present :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r132808084
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionMapping.java ---
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+
+import java.util.*;
+import java.util.function.BiFunction;
+
+public class ExtensionMapping {
+
+ private final Map<String, Map<String, Set>>
extensionNameMap = new HashMap<>();
--- End diff --
So - the goal with the Extension* stuff was to remove the necessity to
statically declare the classes of extensions used. They are passed in to the
ExtensionManager and the BundleThreadContextClassLoader. The ExtensionMapping
is different, it ends up getting the classes from discover in the nifi
properties, where the classes are defined, and typed.
So yes, the type does subdivide the namespace.
Type -> Classname -> coordinates
Parser -> MessageParser -> Parser Extension Coordinates
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 Thanks Matt. I will go through and reply later tonight. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 I will also post the review issues that I do not think have been touched on yet --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r132808147
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,436 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * Scans through the classpath to load all extension components using the
service provider API and
+ * running through all classloaders (root, BUNDLEs).
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+ private static final Logger logger =
LoggerFactory.getLogger(ExtensionManager.class);
+
+ public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new
BundleCoordinates(
+ BundleCoordinates.DEFAULT_GROUP, "system",
BundleCoordinates.DEFAULT_VERSION);
+
+ // Maps a service definition (interface) to those classes that implement
the interface
+ private static final Map<Class, Set> definitionMap = new
HashMap<>();
+
+ private static final Map<String, List> classNameBundleLookup =
new HashMap<>();
+ private static final Map<BundleCoordinates, Bundle>
bundleCoordinateBundleLookup = new HashMap<>();
+ private static final Map<ClassLoader, Bundle> classLoaderBundleLookup =
new HashMap<>();
+
+ private static final Set requiresInstanceClassLoading = new
HashSet<>();
+ private static final Map<String, ClassLoader> instanceClassloaderLookup
= new ConcurrentHashMap<>();
+
+ private static AtomicBoolean inited = new AtomicBoolean(false);
+
+ // should initialize class definitions
+ public static void initClassDefinitions(final List classes) {
+if (classes != null) {
+ for (Class clazz : classes) {
+definitionMap.put(clazz, new HashSet<>());
+ }
+}
+inited.set(true);
+ }
+
+ public static void resetClassDefinitions() {
+definitionMap.clear();
+inited.set(false);
+ }
+
+ /**
+ * Loads all extension class types that can be found on the bootstrap
classloader and by creating
+ * classloaders for all BUNDLES found within the classpath.
+ *
+ * @param bundles the bundles to scan through in search of extensions
+ */
+ public static void discoverExtensions(final Bundle systemBundle, final
Set bundles)
+ throws NotInitializedException {
+checkInitialized();
+// get the current context class loader
+ClassLoader currentContextClassLoader =
Thread.currentThread().getContextClassLoader();
+
+// load the system bundle first so that any extensions found in JARs
directly in lib will be registered as
+// being from the system bundle and not from all the other Bundles
+loadExtensions(systemBun
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r132820984
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -0,0 +1,436 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.bundles;
+
+
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.metron.bundles.bundle.Bundle;
+import org.apache.metron.bundles.bundle.BundleCoordinates;
+import org.apache.metron.bundles.bundle.BundleDetails;
+import org.apache.metron.bundles.util.BundleProperties;
+import org.apache.metron.bundles.util.FileUtils;
+import org.apache.metron.bundles.util.StringUtils;
+import
org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading;
+
+import org.atteo.classindex.ClassIndex;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.reflect.Modifier;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+/**
+ * Scans through the classpath to load all extension components using the
service provider API and
+ * running through all classloaders (root, BUNDLEs).
+ *
+ * @ThreadSafe - is immutable
+ */
+@SuppressWarnings("rawtypes")
+public class ExtensionManager {
+
+ private static final Logger logger =
LoggerFactory.getLogger(ExtensionManager.class);
+
+ public static final BundleCoordinates SYSTEM_BUNDLE_COORDINATE = new
BundleCoordinates(
+ BundleCoordinates.DEFAULT_GROUP, "system",
BundleCoordinates.DEFAULT_VERSION);
+
+ // Maps a service definition (interface) to those classes that implement
the interface
+ private static final Map<Class, Set> definitionMap = new
HashMap<>();
+
+ private static final Map<String, List> classNameBundleLookup =
new HashMap<>();
+ private static final Map<BundleCoordinates, Bundle>
bundleCoordinateBundleLookup = new HashMap<>();
+ private static final Map<ClassLoader, Bundle> classLoaderBundleLookup =
new HashMap<>();
+
+ private static final Set requiresInstanceClassLoading = new
HashSet<>();
+ private static final Map<String, ClassLoader> instanceClassloaderLookup
= new ConcurrentHashMap<>();
+
+ private static AtomicBoolean inited = new AtomicBoolean(false);
+
+ // should initialize class definitions
+ public static void initClassDefinitions(final List classes) {
+if (classes != null) {
+ for (Class clazz : classes) {
+definitionMap.put(clazz, new HashSet<>());
+ }
+}
+inited.set(true);
+ }
+
+ public static void resetClassDefinitions() {
+definitionMap.clear();
+inited.set(false);
+ }
+
+ /**
+ * Loads all extension class types that can be found on the bootstrap
classloader and by creating
+ * classloaders for all BUNDLES found within the classpath.
+ *
+ * @param bundles the bundles to scan through in search of extensions
+ */
+ public static void discoverExtensions(final Bundle systemBundle, final
Set bundles)
+ throws NotInitializedException {
+checkInitialized();
+// get the current context class loader
+ClassLoader currentContextClassLoader =
Thread.currentThread().getContextClassLoader();
+
+// load the system bundle first so that any extensions found in JARs
directly in lib will be registered as
+// being from the system bundle and not from all the other Bundles
+loadExtensions(systemBun
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/530#discussion_r132820993 --- Diff: bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java --- @@ -0,0 +1,436 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.bundles; + + +import org.apache.commons.vfs2.FileObject; +import org.apache.commons.vfs2.FileSystemException; +import org.apache.commons.vfs2.FileSystemManager; +import org.apache.metron.bundles.bundle.Bundle; +import org.apache.metron.bundles.bundle.BundleCoordinates; +import org.apache.metron.bundles.bundle.BundleDetails; +import org.apache.metron.bundles.util.BundleProperties; +import org.apache.metron.bundles.util.FileUtils; +import org.apache.metron.bundles.util.StringUtils; +import org.apache.metron.bundles.annotation.behavior.RequiresInstanceClassLoading; + +import org.atteo.classindex.ClassIndex; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.lang.reflect.Modifier; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicBoolean; + +/** + * Scans through the classpath to load all extension components using the service provider API and + * running through all classloaders (root, BUNDLEs). + * + * @ThreadSafe - is immutable --- End diff -- I agree, and I am working on a refactoring. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r132820881
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/VFSBundleClassLoader.java
---
@@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.bundles;
+
+import java.io.IOException;
+import java.net.URL;
+import java.security.CodeSource;
+import java.security.Permission;
+import java.security.PermissionCollection;
+import java.security.Permissions;
+import java.security.SecureClassLoader;
+import java.security.cert.Certificate;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Enumeration;
+import java.util.Iterator;
+import java.util.List;
+import java.util.jar.Attributes;
+import java.util.jar.Attributes.Name;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.commons.vfs2.FileSystemException;
+import org.apache.commons.vfs2.FileSystemManager;
+import org.apache.commons.vfs2.NameScope;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ *
+ * A ClassLoader for loading BUNDLES (plugin archives). BUNDLEs
are designed to
+ * allow isolating bundles of code (comprising one-or-more
+ * plugin classes and their
+ * dependencies) from other such bundles; this allows for dependencies and
+ * processors that require conflicting, incompatible versions of the same
+ * dependency to run in a single instance of a given process.
+ *
+ *
+ * BundleClassLoader follows the delegation model described in
+ * {@link ClassLoader#findClass(java.lang.String)
ClassLoader.findClass(...)};
+ * classes are first loaded from the parent ClassLoader, and only
if
+ * they cannot be found there does the BundleClassLoader provide a
+ * definition. Specifically, this means that resources are loaded from the
application's
+ * conf
+ * and lib directories first, and if they cannot be found there,
are
+ * loaded from the BUNDLE.
+ *
+ *
+ * The packaging of a BUNDLE is such that it is a ZIP file with the
following
+ * directory structure:
+ *
+ *
+ * +META-INF/
+ * +-- bundled-dependencies/
+ * +-- JAR files
+ * +-- MANIFEST.MF
+ *
+ *
+ *
+ *
+ * The MANIFEST.MF file contains the same information as a typical JAR
file but
+ * also includes two additional bundle properties: {@code Bundle-Id} and
+ * {@code Bundle-Dependency-Id}.
+ *
+ *
+ *
+ * The {@code Bundle-Id} provides a unique identifier for this BUNDLE.
+ *
+ *
+ *
+ * The {@code Bundle-Dependency-Id} is optional. If provided, it indicates
that
+ * this BUNDLE should inherit all of the dependencies of the BUNDLE with
the provided
+ * ID. Often times, the BUNDLE that is depended upon is referred to as the
Parent.
+ * This is because its ClassLoader will be the parent ClassLoader of the
+ * dependent BUNDLE.
+ *
+ *
+ *
+ * If a BUNDLE is built using the Bundles Maven Plugin, the {@code
Bundle-Id} property
+ * will be set to the artifactId of the BUNDLE. The {@code
Bundle-Dependency-Id} will
+ * be set to the artifactId of the BUNDLE that is depended upon. For
example, if
+ * BUNDLE A is defined as such:
+ *
+ *
+ * ...
+ * artifactIdbundle-a/artifactId
+ * packagingbundle/packaging
+ * ...
+ * dependencies
+ * dependency
+ * groupIdgroup/groupId
+ * artifactIdbundle-z/artifactId
+ * typebundle/type
+ * /dependency
+ * /dependencies
+ *
+ *
+ *
+ *
+ *
+ * Then the MANIFEST.MF file that is created for Bundle A will have the
following
+ * properties set:
+ *
+ * {@code {Foo}-Id: bundle-a}
+ * {@code {Foo}-Dependency-Id: bundle-z}
+ *
+ * Where is configurable by BundleProperty META_ID_PREFIX [ default Bundle
]
+ *
+ *
[GitHub] metron pull request #692: METRON-1107 add support for handling epoch dates i...
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/692 METRON-1107 add support for handling epoch dates in seconds or milliseconds This PR adds the capability for the Stellar Date functions which take Long times to handle both seconds and milliseconds, so they just handle epoch time. ## Testing run stellar shell and exercise the Stellar date functions using [epochconverter.com](https://www.epochconverter.com). https://user-images.githubusercontent.com/551/29246163-f7b4b882-7fbe-11e7-9fb7-43892f73eca8.png;> Use both the epoch seconds and millisecond values In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron stellar_time_seconds Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/692.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #692 commit c26b3d807893256d79b0cd786ac82a708ec14498 Author: Otto Fowler <ottobackwa...@gmail.com> Date: 2017-08-13T04:28:05Z add support for handling dates in seconds or milliseconds --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #690: METRON-1091 Package STELLAR shell as stand alone
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/690 METRON-1091 Package STELLAR shell as stand alone This PR adds metron-deployment/packaging/archive, and the metron-stellar-shell module. This module packages a stellar shell environment that allows running stellar on a workstation that is not a dev/build machine, nor is a cluster node. The idea is that a metron 'admin' may want to run stellar commands on his local machine. Remote management ( config_get etc ) and profile would be useful, but have not been included. I do not believe it is realistic for the expected user to open firewall ports or have the 'confs' setup to have these functions work as they would on a metron cluster node. This use case would seem to require REST support of some kind. Maybe a /metron-interface/metron-rest-stellar module. Or maybe when extensions come to stellar we can just do it as an extension. ## Testing - package - take /metron-deployment/packaging/archive/metron-stellar-shell/target/metron-stellar-shell-0.4.1-archive.tar.gz and unpack it in some directory - run it - `stellar it up ## Questions for reviewers - Where should the doc for this go? - What NOTICE or other Files should be in the deployed env? - Do we want to post this as a separate download? - Above questions wrt port opening and configuration ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron stellar_stand_alone Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/690.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #690 commit eab62efeed7188794e5054b2af2ab256a3b6d6fc Author: Otto Fowler <ottobackwa...@gmail.com> Date: 2017-08-09T19:25:33Z Initial pass at packaging stellar shell for stand alone deployment. New archive area under packaging for things packaged as archives New stellar script that doesn't rely on having
[GitHub] metron issue #690: METRON-1091 Package STELLAR shell as stand alone
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/690 Casey, this adds a new packaging subfolder -> archive, and generates a new artifact ( the tar.gz ). It does not effect the current deployment of the stellar shell as a part of metron. I put it in deployment / packaging as default for 2 reasons: 1. If we do include profiler/management etc that makes this a metron thing and not a stellar thing 2. We don't have any packaging outside of deployment Both of those things are up for discussion. an RPM of the shell for RPM deployment to NON metron nodes could be a follow on. --- *Comments from [Reviewable](https://reviewable.io:443/reviews/apache/metron/690#-:-KrW6vGRzz8e1jzn5S1b:b-6c7w3h)* --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #692: METRON-1107 add support for handling epoch dates i...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/692#discussion_r132958368
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/EpochUtils.java
---
@@ -0,0 +1,44 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.stellar.common.utils;
+
+/**
+ * Utility functions for working with numbers that represent different
+ * formats of EPOCH time.
+ */
+public class EpochUtils {
+
+ /**
+ * Ensures returns the passed value as milliseconds from EPOCH if the
value is in seconds.
+ * This is done by looking at the number of digits.
+ * If there are 10, then the value is concidered to be in seconds and
will by
+ * muliplited by 1000.
+ * If there not 10, then the original value will be returned.
+ *
+ *
+ *
+ * @param canidate The Long value to concider
+ * @return A Long value
+ */
+ public static Long ensureEpochMillis(Long canidate) {
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #692: METRON-1107 add support for handling epoch dates i...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/692#discussion_r132958347
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/EpochUtils.java
---
@@ -0,0 +1,44 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.stellar.common.utils;
+
+/**
+ * Utility functions for working with numbers that represent different
+ * formats of EPOCH time.
+ */
+public class EpochUtils {
+
+ /**
+ * Ensures returns the passed value as milliseconds from EPOCH if the
value is in seconds.
+ * This is done by looking at the number of digits.
+ * If there are 10, then the value is concidered to be in seconds and
will by
+ * muliplited by 1000.
+ * If there not 10, then the original value will be returned.
+ *
+ *
+ *
+ * @param canidate The Long value to concider
--- End diff --
:%s/x/y/g
will do the whole file :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #692: METRON-1107 add support for handling epoch dates in secon...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/692 This is due to our time stuff being java Calendar based. We should have a new jira to convert to the new java.time concepts. --- *Comments from [Reviewable](https://reviewable.io:443/reviews/apache/metron/692#-:-KrWFFTDseLKfk-zTl1T:b-j6qvyf)* --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #692: METRON-1107 add support for handling epoch dates in secon...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/692 *[metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/EpochUtils.java, line 41 at r1](https://reviewable.io:443/reviews/apache/metron/692#-KrQxEY26NFRZCW9pxUV-r1-41:-KrWFRZOTUTP31ovphsm:b-s1bb91) ([raw file](https://github.com/apache/metron/blob/186d53ec56ee72281e1ec2dc2de3cc265d4b73d5/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/EpochUtils.java#L41)):* Previously, JonZeolla (JonZeolla) wrote⦠We should handle inputs of length 1-9 as well. So, the 10/13 rule is pretty standard. If we do this, then we know we are doing seconds->millis for current timestamps. for < 10, we don't know if it is millis or timestamps, at least that is my thinking. adding a 0 up to 13 for those numbers is equally as bad as just leaving them no? --- *Comments from [Reviewable](https://reviewable.io:443/reviews/apache/metron/692)* --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #692: METRON-1107 add support for handling epoch dates in secon...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/692 I'm going to close this. We need more discussion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #692: METRON-1107 add support for handling epoch dates i...
Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/692 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #692: METRON-1107 add support for handling epoch dates in secon...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/692 *[metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/DateFunctionsTest.java, line 248 at r1](https://reviewable.io:443/reviews/apache/metron/692#-KrQxEY26NFRZCW9pxUU-r1-248:-KrWRfWh6YcoDTceIyN_:buk7ive) ([raw file](https://github.com/apache/metron/blob/186d53ec56ee72281e1ec2dc2de3cc265d4b73d5/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/DateFunctionsTest.java#L248)):* Previously, JonZeolla (JonZeolla) wrote⦠Suggest adding tests for an epoch time at or before Sunday, September 9, 2001 1:46:39 AM. can I ask why? --- *Comments from [Reviewable](https://reviewable.io:443/reviews/apache/metron/692)* --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #692: METRON-1107 add support for handling epoch dates in secon...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/692 *[metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/EpochUtils.java, line 36 at r1](https://reviewable.io:443/reviews/apache/metron/692#-KrQxEY26NFRZCW9pxUV-r1-36:-KrWRjmbZDyExUXs_95R:b-c58pod) ([raw file](https://github.com/apache/metron/blob/186d53ec56ee72281e1ec2dc2de3cc265d4b73d5/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/EpochUtils.java#L36)):* Previously, JonZeolla (JonZeolla) wrote⦠I wasn't assuming vi, I was assuming more of a `sed`-style formatting, plus you only need the g for your canidate typo because it occurs twice on one line. =) one upper --- *Comments from [Reviewable](https://reviewable.io:443/reviews/apache/metron/692)* --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #641: METRON-539: added HASH function for stellar.
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/641 https://reviewable.io/lgtm.png; height="20" width="61"/> --- *Comments from [Reviewable](https://reviewable.io:443/reviews/apache/metron/641#-:-KrWliAsg4-eqkjCEHxu:bnfp4nl)* --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #278: Metron 451 SerDeUtils - java.lang.ClassNotFoundException:...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/278 What is the status of this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #667: METRON-1061 Add FUZZY_SCORE STELLAR function
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/667#discussion_r133621065
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/TextFunctions.java
---
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
contributor license
+ * agreements. See the NOTICE file distributed with this work for
additional information regarding
+ * copyright ownership. The ASF licenses this file to you under the
Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
the License. You may obtain
+ * a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
ANY KIND, either express
+ * or implied. See the License for the specific language governing
permissions and limitations under
+ * the License.
+ */
+
+package org.apache.metron.stellar.dsl.functions;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Locale;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.text.similarity.FuzzyScore;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.Stellar;
+
+public class TextFunctions {
+
+ @Stellar(name = "GET_AVAILABLE_LANGUAGE_TAGS",
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #681: METRON-1079 Add NaN as a keyword in STELLAR language
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/681 I have added IS_NAN() --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133619468
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -121,33 +109,18 @@ private void unInit() {
* @throws IllegalStateException when already initialized with a given
set of extension directories
* and extensionDirs does not match
*/
- public void init(final FileSystemManager fileSystemManager, final
List extensionsDirs,
+ public static void init(final FileSystemManager fileSystemManager, final
List extensionsDirs,
BundleProperties props)
throws FileSystemException, ClassNotFoundException,
URISyntaxException {
if (extensionsDirs == null || fileSystemManager == null) {
throw new NullPointerException("cannot have empty arguments");
}
-InitContext ic = initContext;
-if (ic == null) {
- synchronized (this) {
-ic = initContext;
-if (ic == null) {
- initContext = load(fileSystemManager, extensionsDirs, props);
- ic = initContext;
-}
- }
-} else {
- boolean matching = CollectionUtils
- .isEqualCollection(initContext.extensionDirs, extensionsDirs);
- if (!matching) {
-throw new IllegalStateException(
-"Cannot reinitialize and extension directories cannot change");
- }
- matching = initContext.properties.match(props);
- if (!matching) {
-throw new IllegalStateException(
-"Cannot re-initialize and properties cannot change");
+synchronized (BundleClassLoaders.class) {
+ if (bundleClassLoaders != null) {
+throw new IllegalStateException("BundleClassloader already
exists");
}
+ bundleClassLoaders = new BundleClassLoaders();
+ initContext = bundleClassLoaders.load(fileSystemManager,
extensionsDirs, props);
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133619456
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/BundleClassLoaders.java ---
@@ -95,19 +91,11 @@ public static BundleClassLoaders getInstance() {
*/
public static void reset() {
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133619481
--- Diff:
bundles-lib/src/main/java/org/apache/metron/bundles/ExtensionManager.java ---
@@ -103,61 +104,52 @@ private ExtensionManager(){}
/**
* @return The singleton instance of the ExtensionManager
*/
- public static ExtensionManager getInstance() {
-ExtensionManager result = extensionManager;
-if (result == null) {
- synchronized (ExtensionManager.class) {
-result = extensionManager;
-if (result == null) {
- extensionManager = new ExtensionManager();
- result = extensionManager;
-}
+ public static ExtensionManager getInstance() throws
NotInitializedException {
+synchronized (ExtensionManager.class) {
+ if (extensionManager == null) {
+throw new NotInitializedException("ExtensionManager not
initialized");
}
+ return extensionManager;
}
-return result;
}
/**
* Uninitializes the ExtensionManager.
* TESTING ONLY
*/
+ @VisibleForTesting
public static void reset() {
synchronized (ExtensionManager.class) {
- getInstance().forgetExtensions();
+ initContext = null;
extensionManager = null;
}
}
- private void forgetExtensions() {
- initContext = null;
- }
-
-
/**
* Loads all extension class types that can be found on the bootstrap
classloader and by creating
* classloaders for all BUNDLES found within the classpath.
*
* @param bundles the bundles to scan through in search of extensions
*/
- public void init(final List classes, final Bundle systemBundle,
final Set bundles)
+ public static void init(final List classes, final Bundle
systemBundle, final Set bundles)
throws NotInitializedException {
-InitContext ic = initContext;
-if (ic == null) {
- synchronized (this) {
-ic = initContext;
-if (ic == null) {
- initContext = discoverExtensions(classes, systemBundle, bundles);
- ic = initContext;
-}
+if (systemBundle == null) {
+ throw new IllegalArgumentException("systemBundle is required");
+}
+
+synchronized (ExtensionManager.class) {
+ if (extensionManager != null) {
+throw new IllegalStateException("ExtensionManager already exists");
}
+ extensionManager = new ExtensionManager();
+ initContext = extensionManager.discoverExtensions(classes,
systemBundle, bundles);
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/530#discussion_r133619506
--- Diff:
bundles-lib/src/test/java/org/apache/metron/bundles/BundleThreadContextClassLoaderTest.java
---
@@ -27,11 +27,21 @@
import org.apache.metron.bundles.bundle.Bundle;
import org.apache.metron.bundles.util.BundleProperties;
import org.apache.metron.bundles.util.FileSystemManagerFactory;
+import org.junit.After;
import org.junit.AfterClass;
import org.junit.Test;
public class BundleThreadContextClassLoaderTest {
+ @AfterClass
+ public static void after() {
+BundleClassLoaders.reset();
+ }
--- End diff --
I hate this static stuff.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #638: METRON-933 New stellar function for regex group ca...
GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/638 METRON-933 New stellar function for regex group capture New Stellar function for capturing a group from a Regex Pattern. I created a new RegExFunctions in Stellar for this, and moved the existing RegEx Match function there. I also created the corresponding test class, and added a test for the existing RegEx Match function as well, since there didn't seem to be one. To support these functions I created a thread local cache for compiled Pattern objects. Testing -> Build, run tests Start stellar shell and execute the functions. ### For all changes: - [x ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ x] Have you included steps or a guide to how the change may be verified and tested manually? - [x ] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [x ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron stellar_regex_capture Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/638.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #638 commit 3d7a79d2afaeb521113f1d77ba6211a0a52e8401 Author: Otto Fowler <ottobackwa...@gmail.com> Date: 2017-07-07T18:52:04Z New stellar function for regex group capture Create a test for existing regex match function Move both to RegExFunctions Create a cache for compiled patterns (thread local) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #652: METRON-1039: Add ZIP function to Stellar
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/652#discussion_r127230597 --- Diff: metron-stellar/stellar-common/README.md --- @@ -711,6 +713,18 @@ In the core language functions, we support basic functional programming primitiv * dateTime - The datetime as a long representing the milliseconds since unix epoch * Returns: The current year +### `ZIP` + * Description: Zips lists into a single list where the ith element is an list containing the ith items from the constituent lists. --- End diff -- That would great --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #642: METRON-984 Stellar functions to decode encoded fields or ...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/642 Will do --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #636: METRON-1022: Elasticsearch REST endpoint
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/636 @cestella https://issues.apache.org/jira/browse/METRON-956 Is the jira I had created on this topic. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/650#discussion_r127220966
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/MathFunctions.java
---
@@ -25,17 +25,39 @@
import org.apache.metron.stellar.dsl.StellarFunction;
import java.util.List;
+import java.util.function.Function;
public class MathFunctions {
- @Stellar(name="ABS"
- ,description="Returns the absolute value of a number."
- ,params = {
-"number - The number to take the absolute value of"
-}
- , returns="The absolute value of the number passed in."
- )
- public static class Abs implements StellarFunction {
+ private enum SingleArgMathFunctions implements Function<Double, Double> {
+ABS(d -> Math.abs(d)),
+CEIL(d -> Math.ceil(d)),
+COS(d -> Math.cos(d)),
+FLOOR(d -> Math.floor(d)),
+LOG10(d -> Math.log10(d)),
+LOG2(d -> Math.log(d)/Math.log(2)),
+LN(d -> Math.log(d)),
+SIN(d -> Math.sin(d)),
+SQRT(d -> Math.sqrt(d)),
+TAN(d -> Math.tan(d)),
+;
+
--- End diff --
Is there a reason why only this subset of Math.* is exposed? What about
the rest? Are they not needed, applicable, or left for future work?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/650#discussion_r127220715
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/MathFunctions.java
---
@@ -25,17 +25,39 @@
import org.apache.metron.stellar.dsl.StellarFunction;
import java.util.List;
+import java.util.function.Function;
public class MathFunctions {
- @Stellar(name="ABS"
- ,description="Returns the absolute value of a number."
- ,params = {
-"number - The number to take the absolute value of"
-}
- , returns="The absolute value of the number passed in."
- )
- public static class Abs implements StellarFunction {
+ private enum SingleArgMathFunctions implements Function<Double, Double> {
--- End diff --
Why have this private? why not have stellar/common/util/math/*
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #652: METRON-1039: Add ZIP function to Stellar
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/652#discussion_r127245708
--- Diff:
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/FunctionalFunctionsTest.java
---
@@ -24,13 +24,124 @@
import org.junit.Assert;
import org.junit.Test;
+import java.util.ArrayList;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import static
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
public class FunctionalFunctionsTest {
@Test
+ public void testZipLongest_boundary() {
+for (String expr : ImmutableList.of( "ZIP_LONGEST()"
+, "ZIP_LONGEST( null, null )"
+, "ZIP_LONGEST( [], null )"
+, "ZIP_LONGEST( [], [] )"
+, "ZIP_LONGEST( null, [] )"
+)
+)
+{
+ List<List> o = (List<List>) run(expr, new
HashMap<>());
+ Assert.assertEquals(0, o.size());
+}
+ }
+
+ @Test
+ public void testZip_longest() {
+Map<String, Object> variables = ImmutableMap.of(
+"list1" , ImmutableList.of(1, 2, 3)
+,"list2", ImmutableList.of(4, 5, 6, 7)
+);
+for (String expr : ImmutableList.of( "ZIP_LONGEST(list1, list2)"
+, "ZIP_LONGEST( [1, 2, 3], [4, 5, 6, 7] )"
+)
+)
+{
+ List<List> o = (List<List>) run(expr, variables);
+ Assert.assertEquals(4, o.size());
+ for (int i = 0; i < 3; ++i) {
+List l = o.get(i);
+Assert.assertEquals(2, l.size());
+Assert.assertEquals(i+1, l.get(0));
+Assert.assertEquals(i+4, l.get(1));
+ }
+ {
+int i = 3;
+List l = o.get(i);
+Assert.assertEquals(2, l.size());
+Assert.assertNull(l.get(0));
+Assert.assertEquals(i+4, l.get(1));
+ }
+}
+
+
+for (String expr : ImmutableList.of(
+ "REDUCE(ZIP_LONGEST(list2, list1), (s, x) -> s + GET_FIRST(x)
* GET_LAST(x), 0)"
+, "REDUCE(ZIP_LONGEST( [1, 2, 3], [4, 5, 6, 7] ), (s, x) -> s
+ GET_FIRST(x) * GET_LAST(x), 0)"
+, "REDUCE(ZIP_LONGEST(list1, list2), (s, x) -> s +
GET_FIRST(x) * GET_LAST(x), 0)" //this works because stellar treats nulls as 0
in arithmetic operations.
+, "REDUCE(ZIP_LONGEST(list1, list2), (s, x) -> s +
(GET_FIRST(x) == null?0:GET_FIRST(x)) * (GET_LAST(x) == null?0:GET_LAST(x)),
0)" //with proper guarding NOT assuming stellar peculiarities
+)
+)
+{
+ int o = (int) run(expr, variables);
+ Assert.assertEquals(1*4 + 2*5 + 3*6, o, 1e-7);
--- End diff --
and don't call him shirly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron pull request #652: METRON-1039: Add ZIP function to Stellar
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/652#discussion_r127245624
--- Diff:
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/FunctionalFunctionsTest.java
---
@@ -24,13 +24,124 @@
import org.junit.Assert;
import org.junit.Test;
+import java.util.ArrayList;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import static
org.apache.metron.stellar.common.utils.StellarProcessorUtils.run;
public class FunctionalFunctionsTest {
@Test
+ public void testZipLongest_boundary() {
+for (String expr : ImmutableList.of( "ZIP_LONGEST()"
+, "ZIP_LONGEST( null, null )"
+, "ZIP_LONGEST( [], null )"
+, "ZIP_LONGEST( [], [] )"
+, "ZIP_LONGEST( null, [] )"
+)
+)
+{
+ List<List> o = (List<List>) run(expr, new
HashMap<>());
+ Assert.assertEquals(0, o.size());
+}
+ }
+
+ @Test
+ public void testZip_longest() {
+Map<String, Object> variables = ImmutableMap.of(
+"list1" , ImmutableList.of(1, 2, 3)
+,"list2", ImmutableList.of(4, 5, 6, 7)
+);
+for (String expr : ImmutableList.of( "ZIP_LONGEST(list1, list2)"
+, "ZIP_LONGEST( [1, 2, 3], [4, 5, 6, 7] )"
+)
+)
+{
+ List<List> o = (List<List>) run(expr, variables);
+ Assert.assertEquals(4, o.size());
+ for (int i = 0; i < 3; ++i) {
+List l = o.get(i);
+Assert.assertEquals(2, l.size());
+Assert.assertEquals(i+1, l.get(0));
+Assert.assertEquals(i+4, l.get(1));
+ }
+ {
+int i = 3;
+List l = o.get(i);
+Assert.assertEquals(2, l.size());
+Assert.assertNull(l.get(0));
+Assert.assertEquals(i+4, l.get(1));
+ }
+}
+
+
+for (String expr : ImmutableList.of(
+ "REDUCE(ZIP_LONGEST(list2, list1), (s, x) -> s + GET_FIRST(x)
* GET_LAST(x), 0)"
+, "REDUCE(ZIP_LONGEST( [1, 2, 3], [4, 5, 6, 7] ), (s, x) -> s
+ GET_FIRST(x) * GET_LAST(x), 0)"
+, "REDUCE(ZIP_LONGEST(list1, list2), (s, x) -> s +
GET_FIRST(x) * GET_LAST(x), 0)" //this works because stellar treats nulls as 0
in arithmetic operations.
+, "REDUCE(ZIP_LONGEST(list1, list2), (s, x) -> s +
(GET_FIRST(x) == null?0:GET_FIRST(x)) * (GET_LAST(x) == null?0:GET_LAST(x)),
0)" //with proper guarding NOT assuming stellar peculiarities
+)
+)
+{
+ int o = (int) run(expr, variables);
+ Assert.assertEquals(1*4 + 2*5 + 3*6, o, 1e-7);
--- End diff --
math fight!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #652: METRON-1039: Add ZIP function to Stellar
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/652 +1 - built + test travis style, ran stellar cli. Nice work @cestella --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #650: METRON-1038: Stellar should have a better collecti...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/650#discussion_r127245372
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/MathFunctions.java
---
@@ -25,17 +25,39 @@
import org.apache.metron.stellar.dsl.StellarFunction;
import java.util.List;
+import java.util.function.Function;
public class MathFunctions {
- @Stellar(name="ABS"
- ,description="Returns the absolute value of a number."
- ,params = {
-"number - The number to take the absolute value of"
-}
- , returns="The absolute value of the number passed in."
- )
- public static class Abs implements StellarFunction {
+ private enum SingleArgMathFunctions implements Function<Double, Double> {
+ABS(d -> Math.abs(d)),
+CEIL(d -> Math.ceil(d)),
+COS(d -> Math.cos(d)),
+FLOOR(d -> Math.floor(d)),
+LOG10(d -> Math.log10(d)),
+LOG2(d -> Math.log(d)/Math.log(2)),
+LN(d -> Math.log(d)),
+SIN(d -> Math.sin(d)),
+SQRT(d -> Math.sqrt(d)),
+TAN(d -> Math.tan(d)),
+;
+
--- End diff --
I agree. I think that in general when we are adapting or exposing a set of
functionality from java or from apache-commons we should state our intent
explicitly for the record.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
