[jira] [Commented] (METRON-1292) Metron should support Apache Nifi Storm Spout
[ https://issues.apache.org/jira/browse/METRON-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244853#comment-16244853 ] Otto Fowler commented on METRON-1292: - [~laurensv] I think this is the jira you are looking for: https://issues.apache.org/jira/browse/METRON-1293 > Metron should support Apache Nifi Storm Spout > - > > Key: METRON-1292 > URL: https://issues.apache.org/jira/browse/METRON-1292 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > https://issues.apache.org/jira/browse/NIFI-606 > Nifi has a storm spout for communication with nifi. > Metron should explore if there is any value to supporting a nifi storm spout > as opposed > to sending to kafka. > Sample : https://github.com/bbende/nifi-storm-topology -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1292) Metron should support Apache Nifi Storm Spout
[ https://issues.apache.org/jira/browse/METRON-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244807#comment-16244807 ] Laurens Vets commented on METRON-1292: -- I can see the value of having a Storm spout sending Metron processed events to NiFi for further processing or storage or whatever else NiFi can provide. I use NiFi a lot to do some preprocessing with for instance AWS CloudTrail logs, so having the option to put events back in NiFi would be a very interesting option to explore. Another option would be to save processed events to S3 for storage without having to write an S3 Storm module. You can replace S3 in the previous sentence by _any_ storage processor NiFi provides. > Metron should support Apache Nifi Storm Spout > - > > Key: METRON-1292 > URL: https://issues.apache.org/jira/browse/METRON-1292 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > https://issues.apache.org/jira/browse/NIFI-606 > Nifi has a storm spout for communication with nifi. > Metron should explore if there is any value to supporting a nifi storm spout > as opposed > to sending to kafka. > Sample : https://github.com/bbende/nifi-storm-topology -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1212) Bundles and Maven Plugin
[ https://issues.apache.org/jira/browse/METRON-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244651#comment-16244651 ] ASF GitHub Bot commented on METRON-1212: Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/774#discussion_r149783542 --- Diff: bundles-maven-plugin/README.md --- @@ -0,0 +1,230 @@ + +# Apache Metron Bundle Maven Plugin + +Apache Metron Bundles Maven Plugin helps to build Bundles Archives to support the classloader isolation model. + +## Table of Contents + +- [Requirements](#requirements) +- [Building](#building) +- [Getting Stared](#getting_started) +- [Getting Help](#getting-help) +- [License](#license) + +## Requirements +* JDK 1.7 or higher +* Apache Maven 3.1.0 or higher + +## Building + +Building the bundles-maven-plugin module should be rare since it will be released infrequently compared to +the main 'metron' code tree. + +- Build with `mvn clean install` +- Presuming you need to make use of changes to the bundles-maven-plugin module, you should next + go to the [metron](../metron) directory and follow its instructions. + +## Getting Started + +While it is most likely +that a maven archetype is being utilized to create bundles, as part of a toolkit etc, you may want to create on manually, or may need to create a project for use in an archetype. + +The plugin is utilized by setting the packaging of a maven module to 'bundle'. + +```xml +bundle +``` + +This means that when you package this module, any of it's non-provided dependencies will be packaged into the produced bundle ( and all of their non-provided dependencies as well). +Since a library may not always be distributed as part of a bundle with all it's dependencies, the bundle module +shall be a separate module from the actual classes and dependencies to be bundled. + +A very simple example layout for a project that utilizes bundles would be: + +```bash +├── README.md +├── pom.xml +├── testapp +│ ├── pom.xml +│ ├── src +│ │ ├── main +│ │ │ └── java +│ │ │ └── org +│ │ │ └── apache +│ │ │ └── test +│ │ │ └── App.java +│ │ └── test +│ │ └── java +│ │ └── org +│ │ └── apache +│ │ └── test +│ │ └── AppTest.java +└── testappbundle +├── pom.xml +``` +Where testappbundle is the bundle module that creates a bundle of testapp, and contains the following pom.xml: +```xml + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + +test.bundles.plugin +org.apache.test +1.0-SNAPSHOT + + 4.0.0 + + test.app.bundle + + + bundle + + + + + org.apache.test + test.app + 1.0-SNAPSHOT + + + + + + + + + org.apache.metron + bundles-maven-plugin + 0.4.2 + true + + + + + + + +org.apache.metron +bundles-maven-plugin +0.4.2 +true + + + + +``` +When the module is packaged, it packages all of it's non-provided dependencies into the bundles /bundled-dependencies directory. +Thus, to create a bundle of a module's jar and that jar's non-provided dependencies, you add that module to your +bundle modules dependencies. You can unzip and examine the bundle in the target directory, and verify +it's contents, which should be similar to : + +```bash +-> % tree . +. +└── META-INF +├── MANIFEST.MF +├── bundled-dependencies +│ ├── log4j-1.2.17.jar +│ ├── metron-common-0.4.1.jar +│ ├── slf4j-api-1.7.7.jar +│ ├── slf4j-log4j12-1.7.7.jar +│ └── test.app-1.0-SNAPSHOT.jar +└── maven +└── org.apache.test +└── test.app.bundle +├── pom.properties +└── pom.xml +``` + +This reflects the testapp project, which has these dependencies : + +```xml + + + org.apache.metron + metron-common + 0.4.1 + + + junit + junit + 3.8.1 + test + + +``` +metron-common
[jira] [Commented] (METRON-1309) Change metron-deployment to pull the plugin from apache/metron-bro-plugin-kafka
[ https://issues.apache.org/jira/browse/METRON-1309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244642#comment-16244642 ] ASF GitHub Bot commented on METRON-1309: GitHub user JonZeolla opened a pull request: https://github.com/apache/metron/pull/837 METRON-1309: Change metron-deployment to pull the plugin from apache/metron-bro-plugin-kafka ## Contributor Comments Based on the conversation on the [mailing list](https://lists.apache.org/thread.html/dd63e5bcf8a288dfaf3cdd551128e3df59580367080bda5f344e8538@%3Cdev.metron.apache.org%3E), this removes the bro kafka plugin from this repo and pulls it from our new apache/metron-bro-plugin-kafka repo. # Testing ## Please hold on testing, this needs to be validated 1. Create a working directory and pull in this PR ``` mkdir ~/metron-1309 git clone https://github.com/apache/metron ~/metron-1309/metron cd ~/metron-1309/metron git remote add jonzeolla https://github.com/jonzeolla/metron git pull jonzeolla METRON-1309 ``` 1. Modify [this](https://github.com/JonZeolla/metron/blob/METRON-1309/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20) to remove `sensors,` (to spin up the real sensors). ``` sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile ``` 1. Spin up full-dev ``` cd metron-deployment/vagrant/full-dev-platform/ vagrant up ``` 1. Set up the environment in full-dev. ``` vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin service monit stop && service sensor-stubs stop bro && broctl stop ``` 1. Monitor the bro kafka topic ``` # Open a new terminal cd ~/metron-1309/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro ``` 1. Run bro against some public pcaps. ``` mkdir -p ~/brotmp/nitroba ~/brotmp/example-traffic wget https://www.bro.org/static/traces/exercise-traffic.pcap -O ~/brotmp/example-traffic/exercise-traffic.pcap wget http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap -O ~/brotmp/nitroba/nitroba.pcap cd ~/brotmp/example-traffic bro -r exercise-traffic.pcap /usr/local/bro/share/bro/site/local.bro -C cd ~/brotmp/nitroba bro -r nitroba.pcap /usr/local/bro/share/bro/site/local.bro -C ``` 1. Verify that logs are properly getting to the bro kafka topic, via the terminal you set up two steps ago. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [X] Have you included steps or a guide to how the change may be verified and tested manually? - [X] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [X] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check
[jira] [Commented] (METRON-1212) Bundles and Maven Plugin
[ https://issues.apache.org/jira/browse/METRON-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244598#comment-16244598 ] ASF GitHub Bot commented on METRON-1212: Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/774#discussion_r149772760 --- Diff: bundles-maven-plugin/README.md --- @@ -0,0 +1,230 @@ + +# Apache Metron Bundle Maven Plugin + +Apache Metron Bundles Maven Plugin helps to build Bundles Archives to support the classloader isolation model. + +## Table of Contents + +- [Requirements](#requirements) +- [Building](#building) +- [Getting Stared](#getting_started) +- [Getting Help](#getting-help) +- [License](#license) + +## Requirements +* JDK 1.7 or higher +* Apache Maven 3.1.0 or higher + +## Building + +Building the bundles-maven-plugin module should be rare since it will be released infrequently compared to +the main 'metron' code tree. + +- Build with `mvn clean install` +- Presuming you need to make use of changes to the bundles-maven-plugin module, you should next + go to the [metron](../metron) directory and follow its instructions. + +## Getting Started + +While it is most likely +that a maven archetype is being utilized to create bundles, as part of a toolkit etc, you may want to create on manually, or may need to create a project for use in an archetype. + +The plugin is utilized by setting the packaging of a maven module to 'bundle'. + +```xml +bundle +``` + +This means that when you package this module, any of it's non-provided dependencies will be packaged into the produced bundle ( and all of their non-provided dependencies as well). +Since a library may not always be distributed as part of a bundle with all it's dependencies, the bundle module +shall be a separate module from the actual classes and dependencies to be bundled. + +A very simple example layout for a project that utilizes bundles would be: + +```bash +├── README.md +├── pom.xml +├── testapp +│ ├── pom.xml +│ ├── src +│ │ ├── main +│ │ │ └── java +│ │ │ └── org +│ │ │ └── apache +│ │ │ └── test +│ │ │ └── App.java +│ │ └── test +│ │ └── java +│ │ └── org +│ │ └── apache +│ │ └── test +│ │ └── AppTest.java +└── testappbundle +├── pom.xml +``` +Where testappbundle is the bundle module that creates a bundle of testapp, and contains the following pom.xml: +```xml + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + +test.bundles.plugin +org.apache.test +1.0-SNAPSHOT + + 4.0.0 + + test.app.bundle + + + bundle + + + + + org.apache.test + test.app + 1.0-SNAPSHOT + + + + + + + + + org.apache.metron + bundles-maven-plugin + 0.4.2 + true + + + + + + + +org.apache.metron +bundles-maven-plugin +0.4.2 +true + + + + +``` +When the module is packaged, it packages all of it's non-provided dependencies into the bundles /bundled-dependencies directory. +Thus, to create a bundle of a module's jar and that jar's non-provided dependencies, you add that module to your +bundle modules dependencies. You can unzip and examine the bundle in the target directory, and verify +it's contents, which should be similar to : + +```bash +-> % tree . +. +└── META-INF +├── MANIFEST.MF +├── bundled-dependencies +│ ├── log4j-1.2.17.jar +│ ├── metron-common-0.4.1.jar +│ ├── slf4j-api-1.7.7.jar +│ ├── slf4j-log4j12-1.7.7.jar +│ └── test.app-1.0-SNAPSHOT.jar +└── maven +└── org.apache.test +└── test.app.bundle +├── pom.properties +└── pom.xml +``` + +This reflects the testapp project, which has these dependencies : + +```xml + + + org.apache.metron + metron-common + 0.4.1 + + + junit + junit + 3.8.1 + test + + +``` +metron-common
[jira] [Created] (METRON-1309) Change metron-deployment to pull the plugin from apache/metron-bro-plugin-kafka
Jon Zeolla created METRON-1309: -- Summary: Change metron-deployment to pull the plugin from apache/metron-bro-plugin-kafka Key: METRON-1309 URL: https://issues.apache.org/jira/browse/METRON-1309 Project: Metron Issue Type: Sub-task Reporter: Jon Zeolla Change the metron-deployment ansible scripts to pull the plugin from apache/metron-bro-plugin-kafka. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1091) STELLAR Shell: Stand Alone installation
[ https://issues.apache.org/jira/browse/METRON-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244562#comment-16244562 ] ASF GitHub Bot commented on METRON-1091: Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/690#discussion_r149767436 --- Diff: metron-stellar/stellar-common/README.md --- @@ -1346,6 +1346,46 @@ Please note that functions are loading lazily in the background and will be unav %functions ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR ``` +## Stellar Shell Stand Alone + +The Stellar Shell is also packaged as a stand alone application. It can be unpacked on any supported +operating system. + +```bash +metron-stellar/stellar-common/target/stellar-common-0.4.1-stand-alone.tar.gz +``` + +When unpacked, the following structure will be created: + +```bash +. +├── bin +│ └── stellar +└── lib +└── stellar-common-0.4.1-uber.jar +``` + +To run the Stellar Shell run the following from the directory you unpacked to: + +```bash +bin/stellar +``` + +```bash +-> % bin/stellar +Stellar, Go! +Please note that functions are loading lazily in the background and will be unavailable until loaded fully. +[Stellar]>>> Functions loaded, you may refer to functions now... + +[Stellar]>>> %functions +ABS, APPEND_IF_MISSING, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CEILING, CHOMP, CHOP, COS, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DECODE, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENCODE, ENDS_WITH, EXP, FILL_LEFT, FILL_RIGHT, FILTER, FLOOR, FORMAT, GET, GET_FIRST, GET_LAST, GET_SUPPORTED_ENCODINGS, IN_SUBNET, IS_EMPTY, IS_ENCODING, JOIN, LENGTH, LIST_ADD, LN, LOG10, LOG2, MAP, MAP_EXISTS, MAP_GET, MONTH, PREPEND_IF_MISSING, REDUCE, REGEXP_GROUP_VAL, REGEXP_MATCH, ROUND, SIN, SPLIT, SQRT, STARTS_WITH, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TAN, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR, ZIP, ZIP_LONGEST +[Stellar]>>> +``` + +By default the shell will have the base Stellar Language commands available. Any jars in the lib directory +that contain Stellar functions will also be loaded, and their commands will be available to shell, as long +as their dependencies are satisfied. --- End diff -- What about if you just change it from: have the base Stellar Language commands to have only the base Stellar Language commands ? I'm not looking to explicitly mention *what* functions would not exist here, and why, but simply make it more obvious that things will be missing here by default. > STELLAR Shell: Stand Alone installation > --- > > Key: METRON-1091 > URL: https://issues.apache.org/jira/browse/METRON-1091 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > Labels: shell, stellar > > We should have a way that the stellar shell can be installed and run easily > on any machine that supports java. > This may just be a new UBER Jar, or something else. > It may also be useful to have this app start up and initialize a zookeeper > instance and other things for a testing or simulated environment. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1091) STELLAR Shell: Stand Alone installation
[ https://issues.apache.org/jira/browse/METRON-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244558#comment-16244558 ] ASF GitHub Bot commented on METRON-1091: Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/690#discussion_r149766318 --- Diff: metron-stellar/stellar-common/README.md --- @@ -1346,6 +1346,46 @@ Please note that functions are loading lazily in the background and will be unav %functions ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR ``` +## Stellar Shell Stand Alone + +The Stellar Shell is also packaged as a stand alone application. It can be unpacked on any supported +operating system. + +```bash +metron-stellar/stellar-common/target/stellar-common-0.4.1-stand-alone.tar.gz +``` + +When unpacked, the following structure will be created: + +```bash +. +├── bin +│ └── stellar +└── lib +└── stellar-common-0.4.1-uber.jar +``` + +To run the Stellar Shell run the following from the directory you unpacked to: + +```bash +bin/stellar +``` + +```bash +-> % bin/stellar +Stellar, Go! +Please note that functions are loading lazily in the background and will be unavailable until loaded fully. +[Stellar]>>> Functions loaded, you may refer to functions now... + +[Stellar]>>> %functions +ABS, APPEND_IF_MISSING, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CEILING, CHOMP, CHOP, COS, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DECODE, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENCODE, ENDS_WITH, EXP, FILL_LEFT, FILL_RIGHT, FILTER, FLOOR, FORMAT, GET, GET_FIRST, GET_LAST, GET_SUPPORTED_ENCODINGS, IN_SUBNET, IS_EMPTY, IS_ENCODING, JOIN, LENGTH, LIST_ADD, LN, LOG10, LOG2, MAP, MAP_EXISTS, MAP_GET, MONTH, PREPEND_IF_MISSING, REDUCE, REGEXP_GROUP_VAL, REGEXP_MATCH, ROUND, SIN, SPLIT, SQRT, STARTS_WITH, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TAN, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR, ZIP, ZIP_LONGEST +[Stellar]>>> +``` + +By default the shell will have the base Stellar Language commands available. Any jars in the lib directory +that contain Stellar functions will also be loaded, and their commands will be available to shell, as long +as their dependencies are satisfied. --- End diff -- I agree that this could be more clear, but I don't think this is the PR for it. I think in general we need to either document or provide some more information through the functions on what should be included. IE: Move the documentation so that the stellar/README only has things in stellar-common -or- decorate which lib needs to be included with each function in the doc etc. > STELLAR Shell: Stand Alone installation > --- > > Key: METRON-1091 > URL: https://issues.apache.org/jira/browse/METRON-1091 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > Labels: shell, stellar > > We should have a way that the stellar shell can be installed and run easily > on any machine that supports java. > This may just be a new UBER Jar, or something else. > It may also be useful to have this app start up and initialize a zookeeper > instance and other things for a testing or simulated environment. -- This message was sent by Atlassian
[jira] [Commented] (METRON-1091) STELLAR Shell: Stand Alone installation
[ https://issues.apache.org/jira/browse/METRON-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244532#comment-16244532 ] ASF GitHub Bot commented on METRON-1091: Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/690#discussion_r149686086 --- Diff: metron-stellar/stellar-common/README.md --- @@ -1346,6 +1346,46 @@ Please note that functions are loading lazily in the background and will be unav %functions ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR ``` +## Stellar Shell Stand Alone + +The Stellar Shell is also packaged as a stand alone application. It can be unpacked on any supported +operating system. + +```bash +metron-stellar/stellar-common/target/stellar-common-0.4.1-stand-alone.tar.gz --- End diff -- `s/0\.4\.1/0.4.2/` > STELLAR Shell: Stand Alone installation > --- > > Key: METRON-1091 > URL: https://issues.apache.org/jira/browse/METRON-1091 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > Labels: shell, stellar > > We should have a way that the stellar shell can be installed and run easily > on any machine that supports java. > This may just be a new UBER Jar, or something else. > It may also be useful to have this app start up and initialize a zookeeper > instance and other things for a testing or simulated environment. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1091) STELLAR Shell: Stand Alone installation
[ https://issues.apache.org/jira/browse/METRON-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244530#comment-16244530 ] ASF GitHub Bot commented on METRON-1091: Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/690#discussion_r149759730 --- Diff: metron-stellar/stellar-common/README.md --- @@ -1346,6 +1346,46 @@ Please note that functions are loading lazily in the background and will be unav %functions ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR ``` +## Stellar Shell Stand Alone + +The Stellar Shell is also packaged as a stand alone application. It can be unpacked on any supported +operating system. + +```bash +metron-stellar/stellar-common/target/stellar-common-0.4.1-stand-alone.tar.gz +``` + +When unpacked, the following structure will be created: + +```bash +. +├── bin +│ └── stellar +└── lib +└── stellar-common-0.4.1-uber.jar +``` + +To run the Stellar Shell run the following from the directory you unpacked to: + +```bash +bin/stellar +``` + +```bash +-> % bin/stellar +Stellar, Go! +Please note that functions are loading lazily in the background and will be unavailable until loaded fully. +[Stellar]>>> Functions loaded, you may refer to functions now... + +[Stellar]>>> %functions +ABS, APPEND_IF_MISSING, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CEILING, CHOMP, CHOP, COS, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DECODE, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENCODE, ENDS_WITH, EXP, FILL_LEFT, FILL_RIGHT, FILTER, FLOOR, FORMAT, GET, GET_FIRST, GET_LAST, GET_SUPPORTED_ENCODINGS, IN_SUBNET, IS_EMPTY, IS_ENCODING, JOIN, LENGTH, LIST_ADD, LN, LOG10, LOG2, MAP, MAP_EXISTS, MAP_GET, MONTH, PREPEND_IF_MISSING, REDUCE, REGEXP_GROUP_VAL, REGEXP_MATCH, ROUND, SIN, SPLIT, SQRT, STARTS_WITH, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TAN, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR, ZIP, ZIP_LONGEST +[Stellar]>>> +``` + +By default the shell will have the base Stellar Language commands available. Any jars in the lib directory +that contain Stellar functions will also be loaded, and their commands will be available to shell, as long +as their dependencies are satisfied. --- End diff -- Maybe we can be a little bit more explicit that, by default, not all stellar functions will work in the standalone shell. For instance, clarifying that only functions available listed in response to %functions will be included may be enough? I quickly noted that BIN, HLLP_*, STATS_*, OUTLIER_MAD_*, etc. weren't in there as profiler/management commands aren't included. Maybe we say that as well? > STELLAR Shell: Stand Alone installation > --- > > Key: METRON-1091 > URL: https://issues.apache.org/jira/browse/METRON-1091 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > Labels: shell, stellar > > We should have a way that the stellar shell can be installed and run easily > on any machine that supports java. > This may just be a new UBER Jar, or something else. > It may also be useful to have this app start up and initialize a zookeeper > instance and other things for a testing or simulated environment. -- This message was sent by Atlassian
[jira] [Commented] (METRON-1091) STELLAR Shell: Stand Alone installation
[ https://issues.apache.org/jira/browse/METRON-1091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244531#comment-16244531 ] ASF GitHub Bot commented on METRON-1091: Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/690#discussion_r149686111 --- Diff: metron-stellar/stellar-common/README.md --- @@ -1346,6 +1346,46 @@ Please note that functions are loading lazily in the background and will be unav %functions ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR ``` +## Stellar Shell Stand Alone + +The Stellar Shell is also packaged as a stand alone application. It can be unpacked on any supported +operating system. + +```bash +metron-stellar/stellar-common/target/stellar-common-0.4.1-stand-alone.tar.gz +``` + +When unpacked, the following structure will be created: + +```bash +. +├── bin +│ └── stellar +└── lib +└── stellar-common-0.4.1-uber.jar --- End diff -- `s/0\.4\.1/0.4.2/` > STELLAR Shell: Stand Alone installation > --- > > Key: METRON-1091 > URL: https://issues.apache.org/jira/browse/METRON-1091 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > Labels: shell, stellar > > We should have a way that the stellar shell can be installed and run easily > on any machine that supports java. > This may just be a new UBER Jar, or something else. > It may also be useful to have this app start up and initialize a zookeeper > instance and other things for a testing or simulated environment. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1212) Bundles and Maven Plugin
[ https://issues.apache.org/jira/browse/METRON-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244480#comment-16244480 ] ASF GitHub Bot commented on METRON-1212: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/774#discussion_r149755588 --- Diff: bundles-maven-plugin/README.md --- @@ -0,0 +1,230 @@ + +# Apache Metron Bundle Maven Plugin + +Apache Metron Bundles Maven Plugin helps to build Bundles Archives to support the classloader isolation model. + +## Table of Contents + +- [Requirements](#requirements) +- [Building](#building) +- [Getting Stared](#getting_started) +- [Getting Help](#getting-help) +- [License](#license) + +## Requirements +* JDK 1.7 or higher +* Apache Maven 3.1.0 or higher + +## Building + +Building the bundles-maven-plugin module should be rare since it will be released infrequently compared to +the main 'metron' code tree. + +- Build with `mvn clean install` +- Presuming you need to make use of changes to the bundles-maven-plugin module, you should next + go to the [metron](../metron) directory and follow its instructions. + +## Getting Started + +While it is most likely +that a maven archetype is being utilized to create bundles, as part of a toolkit etc, you may want to create on manually, or may need to create a project for use in an archetype. + +The plugin is utilized by setting the packaging of a maven module to 'bundle'. + +```xml +bundle +``` + +This means that when you package this module, any of it's non-provided dependencies will be packaged into the produced bundle ( and all of their non-provided dependencies as well). +Since a library may not always be distributed as part of a bundle with all it's dependencies, the bundle module +shall be a separate module from the actual classes and dependencies to be bundled. + +A very simple example layout for a project that utilizes bundles would be: + +```bash +├── README.md +├── pom.xml +├── testapp +│ ├── pom.xml +│ ├── src +│ │ ├── main +│ │ │ └── java +│ │ │ └── org +│ │ │ └── apache +│ │ │ └── test +│ │ │ └── App.java +│ │ └── test +│ │ └── java +│ │ └── org +│ │ └── apache +│ │ └── test +│ │ └── AppTest.java +└── testappbundle +├── pom.xml +``` +Where testappbundle is the bundle module that creates a bundle of testapp, and contains the following pom.xml: +```xml + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + +test.bundles.plugin +org.apache.test +1.0-SNAPSHOT + + 4.0.0 + + test.app.bundle + + + bundle + + + + + org.apache.test + test.app + 1.0-SNAPSHOT + + + + + + + + + org.apache.metron + bundles-maven-plugin + 0.4.2 + true + + + + + + + +org.apache.metron +bundles-maven-plugin +0.4.2 +true + + + + +``` +When the module is packaged, it packages all of it's non-provided dependencies into the bundles /bundled-dependencies directory. +Thus, to create a bundle of a module's jar and that jar's non-provided dependencies, you add that module to your +bundle modules dependencies. You can unzip and examine the bundle in the target directory, and verify +it's contents, which should be similar to : + +```bash +-> % tree . +. +└── META-INF +├── MANIFEST.MF +├── bundled-dependencies +│ ├── log4j-1.2.17.jar +│ ├── metron-common-0.4.1.jar +│ ├── slf4j-api-1.7.7.jar +│ ├── slf4j-log4j12-1.7.7.jar +│ └── test.app-1.0-SNAPSHOT.jar +└── maven +└── org.apache.test +└── test.app.bundle +├── pom.properties +└── pom.xml +``` + +This reflects the testapp project, which has these dependencies : + +```xml + + + org.apache.metron + metron-common + 0.4.1 + + + junit + junit + 3.8.1 + test + + +``` +metron-common
[jira] [Commented] (METRON-1308) Fix Metron Documentation
[ https://issues.apache.org/jira/browse/METRON-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244178#comment-16244178 ] ASF GitHub Bot commented on METRON-1308: Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/836#discussion_r149706098 --- Diff: metron-stellar/stellar-common/README.md --- @@ -257,7 +258,7 @@ In the core language functions, we support basic functional programming primitiv * Description: Adds an element to the bloom filter passed in * Input: * bloom - The bloom filter -* value* - The values to add +* value\*? - The values to add --- End diff -- Okay, I get your intent now. It was being suppressed before because it wasn't escaped, I will try to make it more obvious. > Fix Metron Documentation > > > Key: METRON-1308 > URL: https://issues.apache.org/jira/browse/METRON-1308 > Project: Metron > Issue Type: Bug >Reporter: Jon Zeolla > > I am combing through Metron documentation and I will have various updates for > minor fixes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1308) Fix Metron Documentation
[ https://issues.apache.org/jira/browse/METRON-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244180#comment-16244180 ] ASF GitHub Bot commented on METRON-1308: Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/836#discussion_r149706457 --- Diff: metron-stellar/stellar-common/README.md --- @@ -748,7 +728,7 @@ In the core language functions, we support basic functional programming primitiv ### `MULTISET_INIT` * Description: Creates an empty multiset, which is a map associating objects to their instance counts. * Input: -* input? - An initialization of the multiset +* input?? - An initialization of the multiset --- End diff -- Gotcha, okay. I think I would prefer something like (optional), what do you think? > Fix Metron Documentation > > > Key: METRON-1308 > URL: https://issues.apache.org/jira/browse/METRON-1308 > Project: Metron > Issue Type: Bug >Reporter: Jon Zeolla > > I am combing through Metron documentation and I will have various updates for > minor fixes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1308) Fix Metron Documentation
[ https://issues.apache.org/jira/browse/METRON-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244174#comment-16244174 ] ASF GitHub Bot commented on METRON-1308: Github user cestella commented on the issue: https://github.com/apache/metron/pull/836 So, answering your questions en-masse we use the following similar to how they exist in regex: * `*` - `0` or `n` instances * `?` - `0` or `1` instance * `+` - `1` or `n` instances In retrospect, this may not be as obvious as it seemed at the time. Think we need a key that indicates this? ;) > Fix Metron Documentation > > > Key: METRON-1308 > URL: https://issues.apache.org/jira/browse/METRON-1308 > Project: Metron > Issue Type: Bug >Reporter: Jon Zeolla > > I am combing through Metron documentation and I will have various updates for > minor fixes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1308) Fix Metron Documentation
[ https://issues.apache.org/jira/browse/METRON-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244170#comment-16244170 ] ASF GitHub Bot commented on METRON-1308: Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/836#discussion_r149704557 --- Diff: metron-stellar/stellar-common/README.md --- @@ -748,7 +728,7 @@ In the core language functions, we support basic functional programming primitiv ### `MULTISET_INIT` * Description: Creates an empty multiset, which is a map associating objects to their instance counts. * Input: -* input? - An initialization of the multiset +* input?? - An initialization of the multiset --- End diff -- ? implies it may be included or not, like regex > Fix Metron Documentation > > > Key: METRON-1308 > URL: https://issues.apache.org/jira/browse/METRON-1308 > Project: Metron > Issue Type: Bug >Reporter: Jon Zeolla > > I am combing through Metron documentation and I will have various updates for > minor fixes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1308) Fix Metron Documentation
[ https://issues.apache.org/jira/browse/METRON-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244142#comment-16244142 ] ASF GitHub Bot commented on METRON-1308: Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/836#discussion_r149701086 --- Diff: metron-stellar/stellar-common/README.md --- @@ -257,7 +258,7 @@ In the core language functions, we support basic functional programming primitiv * Description: Adds an element to the bloom filter passed in * Input: * bloom - The bloom filter -* value* - The values to add +* value\*? - The values to add --- End diff -- Why does this have an \*? > Fix Metron Documentation > > > Key: METRON-1308 > URL: https://issues.apache.org/jira/browse/METRON-1308 > Project: Metron > Issue Type: Bug >Reporter: Jon Zeolla > > I am combing through Metron documentation and I will have various updates for > minor fixes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1308) Fix Metron Documentation
[ https://issues.apache.org/jira/browse/METRON-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244166#comment-16244166 ] ASF GitHub Bot commented on METRON-1308: Github user cestella commented on a diff in the pull request: https://github.com/apache/metron/pull/836#discussion_r149703967 --- Diff: metron-stellar/stellar-common/README.md --- @@ -257,7 +258,7 @@ In the core language functions, we support basic functional programming primitiv * Description: Adds an element to the bloom filter passed in * Input: * bloom - The bloom filter -* value* - The values to add +* value\*? - The values to add --- End diff -- It's a repetition symbol, so it implies you can have any number of values. > Fix Metron Documentation > > > Key: METRON-1308 > URL: https://issues.apache.org/jira/browse/METRON-1308 > Project: Metron > Issue Type: Bug >Reporter: Jon Zeolla > > I am combing through Metron documentation and I will have various updates for > minor fixes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1308) Fix Metron Documentation
[ https://issues.apache.org/jira/browse/METRON-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244143#comment-16244143 ] ASF GitHub Bot commented on METRON-1308: Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/836#discussion_r149701209 --- Diff: metron-stellar/stellar-common/README.md --- @@ -849,7 +829,7 @@ In the core language functions, we support basic functional programming primitiv ### `SET_INIT` * Description: Creates an new set * Input: -* input? - An initialization of the set +* input?? - An initialization of the set --- End diff -- Why does this have a \?? > Fix Metron Documentation > > > Key: METRON-1308 > URL: https://issues.apache.org/jira/browse/METRON-1308 > Project: Metron > Issue Type: Bug >Reporter: Jon Zeolla > > I am combing through Metron documentation and I will have various updates for > minor fixes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1308) Fix Metron Documentation
[ https://issues.apache.org/jira/browse/METRON-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244141#comment-16244141 ] ASF GitHub Bot commented on METRON-1308: Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/836#discussion_r149701164 --- Diff: metron-stellar/stellar-common/README.md --- @@ -748,7 +728,7 @@ In the core language functions, we support basic functional programming primitiv ### `MULTISET_INIT` * Description: Creates an empty multiset, which is a map associating objects to their instance counts. * Input: -* input? - An initialization of the multiset +* input?? - An initialization of the multiset --- End diff -- Why does this have a \?? > Fix Metron Documentation > > > Key: METRON-1308 > URL: https://issues.apache.org/jira/browse/METRON-1308 > Project: Metron > Issue Type: Bug >Reporter: Jon Zeolla > > I am combing through Metron documentation and I will have various updates for > minor fixes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1308) Fix Metron Documentation
[ https://issues.apache.org/jira/browse/METRON-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244144#comment-16244144 ] ASF GitHub Bot commented on METRON-1308: Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/836#discussion_r149701286 --- Diff: metron-stellar/stellar-common/README.md --- @@ -1034,15 +1035,15 @@ In the core language functions, we support basic functional programming primitiv See [python](https://docs.python.org/3/library/functions.html#zip) and [wikipedia](https://en.wikipedia.org/wiki/Convolution_(computer_science)) for more context. * Input: -* list* - Lists to zip. +* list\*? - Lists to zip. * Returns: The zip of the lists. The returned list is the min size of all the lists. e.g. `ZIP( [ 1, 2 ], [ 3, 4, 5] ) == [ [1, 3], [2, 4] ]` ### `ZIP_LONGEST` * Description: Zips lists into a single list where the ith element is an list containing the ith items from the constituent lists. See [python](https://docs.python.org/3/library/itertools.html#itertools.zip_longest) and [wikipedia](https://en.wikipedia.org/wiki/Convolution_(computer_science)) for more context. * Input: -* list* - Lists to zip. +* list\*? - Lists to zip. --- End diff -- Why does this have an \*? > Fix Metron Documentation > > > Key: METRON-1308 > URL: https://issues.apache.org/jira/browse/METRON-1308 > Project: Metron > Issue Type: Bug >Reporter: Jon Zeolla > > I am combing through Metron documentation and I will have various updates for > minor fixes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1308) Fix Metron Documentation
[ https://issues.apache.org/jira/browse/METRON-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244145#comment-16244145 ] ASF GitHub Bot commented on METRON-1308: Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/836#discussion_r149701260 --- Diff: metron-stellar/stellar-common/README.md --- @@ -1034,15 +1035,15 @@ In the core language functions, we support basic functional programming primitiv See [python](https://docs.python.org/3/library/functions.html#zip) and [wikipedia](https://en.wikipedia.org/wiki/Convolution_(computer_science)) for more context. * Input: -* list* - Lists to zip. +* list\*? - Lists to zip. --- End diff -- Why does this have an \*? > Fix Metron Documentation > > > Key: METRON-1308 > URL: https://issues.apache.org/jira/browse/METRON-1308 > Project: Metron > Issue Type: Bug >Reporter: Jon Zeolla > > I am combing through Metron documentation and I will have various updates for > minor fixes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1308) Fix Metron Documentation
[ https://issues.apache.org/jira/browse/METRON-1308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244138#comment-16244138 ] ASF GitHub Bot commented on METRON-1308: GitHub user JonZeolla opened a pull request: https://github.com/apache/metron/pull/836 DO NOT MERGE METRON-1308: Fix Metron Documentation ## Contributor Comments More documentation fixes - work in progress but looking to get a review of some findings (see comments) ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JonZeolla/metron METRON-1308 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/836.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #836 commit 3f55bc61a15e9a1406af220181f7e022e593e19d Author: Jon ZeollaDate: 2017-11-08T15:20:23Z METRON-1308 commit 89d12b46dcf75fc514c56f8fa103e65ee044f504 Author: Jon Zeolla Date: 2017-11-08T15:20:33Z Merge branch 'master' of https://github.com/apache/metron into METRON-1308 commit 6e4b2f50ed3ab0c69095b144a3c1b9f89ac92c0c Author: Jon Zeolla Date: 2017-11-08T15:21:44Z METRON-1308 > Fix Metron Documentation > > > Key: METRON-1308 > URL: https://issues.apache.org/jira/browse/METRON-1308 > Project: Metron > Issue Type: Bug >Reporter: Jon Zeolla > > I am combing through Metron documentation and I will have various updates for > minor fixes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (METRON-1308) Fix Metron Documentation
Jon Zeolla created METRON-1308: -- Summary: Fix Metron Documentation Key: METRON-1308 URL: https://issues.apache.org/jira/browse/METRON-1308 Project: Metron Issue Type: Bug Reporter: Jon Zeolla I am combing through Metron documentation and I will have various updates for minor fixes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1212) Bundles and Maven Plugin
[ https://issues.apache.org/jira/browse/METRON-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244105#comment-16244105 ] ASF GitHub Bot commented on METRON-1212: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/774 I checkout your PR and then attempt to do an install and the build fails. ``` $ mvn clean install [INFO] Scanning for projects... Downloading: https://repo.maven.apache.org/maven2/org/apache/metron/bundles-maven-plugin/0.4.2/bundles-maven-plugin-0.4.2.pom [WARNING] The POM for org.apache.metron:bundles-maven-plugin:jar:0.4.2 is missing, no dependency information available Downloading: https://repo.maven.apache.org/maven2/org/apache/metron/bundles-maven-plugin/0.4.2/bundles-maven-plugin-0.4.2.jar [ERROR] [ERROR] Some problems were encountered while processing the POMs: [ERROR] Unresolveable build extension: Plugin org.apache.metron:bundles-maven-plugin:0.4.2 or one of its dependencies could not be resolved: Could not find artifact org.apache.metron:bundles-maven-plugin:jar:0.4.2 in central (https://repo.maven.apache.org/maven2) @ @ [ERROR] The build could not read 1 project -> [Help 1] [ERROR] [ERROR] The project org.apache.metron:bundles-testing:0.4.2 (/Users/nallen/tmp/metron-pr774/bundles-testing/pom.xml) has 1 error [ERROR] Unresolveable build extension: Plugin org.apache.metron:bundles-maven-plugin:0.4.2 or one of its dependencies could not be resolved: Could not find artifact org.apache.metron:bundles-maven-plugin:jar:0.4.2 in central (https://repo.maven.apache.org/maven2) -> [Help 2] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException [ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/PluginManagerException ``` I noticed that you changed the .travis build instructions so that you have to manually go into the `bundles-maven-plugin` project and install that first. ``` $ git diff origin/master -- .travis.yml diff --git a/.travis.yml b/.travis.yml index f5edfb27..8023830a 100644 --- a/.travis.yml +++ b/.travis.yml @@ -33,6 +33,7 @@ before_install: - npm config set prefix $HOME/.npm-prefix --global install: + - cd bundles-maven-plugin && mvn -q install && cd .. - time mvn -q -T 2C -DskipTests clean install script: ``` Ideally we need to be able to just build Metron by running an install from the root directory. Is there no way around this? What troubles did you run into that caused you to have to do this? > Bundles and Maven Plugin > > > Key: METRON-1212 > URL: https://issues.apache.org/jira/browse/METRON-1212 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler > > The first effort will be to land the bundle system and supporting maven > plugin on master -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1289) Alert fields are lost when a MetaAlert is created
[ https://issues.apache.org/jira/browse/METRON-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244009#comment-16244009 ] ASF GitHub Bot commented on METRON-1289: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/824 Do we know why the partial update doesn't work? I'm not necessarily opposed to doing this as a short term fix, but I'd like to know root cause. It seems like overkill to submit a full object update on every alert that goes into a metaalert (e.g. that could be a hundred+ full document reindexes happening every time) > Alert fields are lost when a MetaAlert is created > - > > Key: METRON-1289 > URL: https://issues.apache.org/jira/browse/METRON-1289 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman > > When a MetaAlert is created, the included results are being updated > incorrectly with only the "metaalert" field. This causes subsequent findOne > operations to only return the "metaalert field for that alert. All fields > should continue to be present. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1303) Reorganize the metron-bro-plugin-kafka
[ https://issues.apache.org/jira/browse/METRON-1303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243991#comment-16243991 ] ASF GitHub Bot commented on METRON-1303: Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/1 Holding off on committing this until I hear back on my [suggested roadmap outline](https://lists.apache.org/thread.html/1e93ec4f8f126610ba359c44982bc512a338728ccdf21cebdc53d117@%3Cdev.metron.apache.org%3E). > Reorganize the metron-bro-plugin-kafka > -- > > Key: METRON-1303 > URL: https://issues.apache.org/jira/browse/METRON-1303 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla > > Currently the metron-bro-plugin-kafka plugin gets installed as Bro::Kafka, > which is somewhat confusing based on the way bro packages are currently > named/sorted[1]. Based on this, we should align the plugin namespace to run > under the owner of the repo, which is apache. > 1: https://github.com/bro/packages -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (METRON-914) Build Metron failure
[ https://issues.apache.org/jira/browse/METRON-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243841#comment-16243841 ] Bugra edited comment on METRON-914 at 11/8/17 12:59 PM: Hello, I do have same problem on Centos 7. Is there any update about the topic? ? Is there any other method to get the file or solve the problem manually only for metron-config ? For your information, following this guide: https://cwiki.apache.org/confluence/display/METRON/Metron+0.4.1+with+HDP+2.5+bare-metal+install+on+Centos+7+with+MariaDB+for+Metron+REST Facing problem on this step : Build Metron with HDP 2.5 profile: cd metron {color:red}mvn clean package -DskipTests -T 2C -P HDP-2.5.0.0,mpack{color} was (Author: bugra.sertcelik): Hello, I do have same problem on Centos 7. Is there any update about the topic? ? Is there any other method to get the file or solve the problem manually only for metron-config ? > Build Metron failure > > > Key: METRON-914 > URL: https://issues.apache.org/jira/browse/METRON-914 > Project: Metron > Issue Type: Bug >Affects Versions: 0.4.0 > Environment: Ubuntu 16.04.1 >Reporter: GS Peter > Fix For: 0.4.0 > > Attachments: build.log.tar.gz, build.test.log.gz, npm-debug.log > > Original Estimate: 504h > Remaining Estimate: 504h > > [INFO] metron-config .. FAILURE [04:08 > min] > [INFO] metron-rest-client . SKIPPED > [INFO] metron-rest SKIPPED > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 58:04 min > [INFO] Finished at: 2017-04-28T15:57:16+07:00 > [INFO] Final Memory: 141M/588M > [INFO] > > [ERROR] Failed to execute goal > com.github.eirslett:frontend-maven-plugin:1.3:npm (npm install) on project > metron-config: Failed to run task: 'npm install' failed. (error code 1) -> > [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn -rf :metron-config -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-914) Build Metron failure
[ https://issues.apache.org/jira/browse/METRON-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243841#comment-16243841 ] Bugra commented on METRON-914: -- Hello, I do have same problem on Centos 7. Is there any update about the topic? ? Is there any other method to get the file or solve the problem manually only for metron-config ? > Build Metron failure > > > Key: METRON-914 > URL: https://issues.apache.org/jira/browse/METRON-914 > Project: Metron > Issue Type: Bug >Affects Versions: 0.4.0 > Environment: Ubuntu 16.04.1 >Reporter: GS Peter > Fix For: 0.4.0 > > Attachments: build.log.tar.gz, build.test.log.gz, npm-debug.log > > Original Estimate: 504h > Remaining Estimate: 504h > > [INFO] metron-config .. FAILURE [04:08 > min] > [INFO] metron-rest-client . SKIPPED > [INFO] metron-rest SKIPPED > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 58:04 min > [INFO] Finished at: 2017-04-28T15:57:16+07:00 > [INFO] Final Memory: 141M/588M > [INFO] > > [ERROR] Failed to execute goal > com.github.eirslett:frontend-maven-plugin:1.3:npm (npm install) on project > metron-config: Failed to run task: 'npm install' failed. (error code 1) -> > [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn -rf :metron-config -- This message was sent by Atlassian JIRA (v6.4.14#64029)