[jira] [Commented] (METRON-942) REST Support for Parser Extensions
[ https://issues.apache.org/jira/browse/METRON-942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143162#comment-16143162 ] ASF GitHub Bot commented on METRON-942: --- Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/580 > REST Support for Parser Extensions > -- > > Key: METRON-942 > URL: https://issues.apache.org/jira/browse/METRON-942 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > If and after METRON-777 lands, support will be required for installing parser > extensions into the system, as well as uninstalling them. > The first step in this is to have support in metron-rest for this. > The rest interface should support installing a parser extension from the > assembly tar.gz produced by the archetype. > The result of the installation should be: > * the bundle is installed in the alternate extension library location in hdfs > * the patterns for all parsers in the extension should be deployed to hdfs > * the configurations for each pattern should be installed into ZK ( parser, > indexing, enrichment ) > * an extension configuration, that includes the parsers it produces, and the > default configurations ( the above and elasticsearch etc ) should be entered > into ZK > The reading of these configurations should also be supported. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-942) REST Support for Parser Extensions
[ https://issues.apache.org/jira/browse/METRON-942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143161#comment-16143161 ] ASF GitHub Bot commented on METRON-942: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/580 I am closing this in preparation for feature branch > REST Support for Parser Extensions > -- > > Key: METRON-942 > URL: https://issues.apache.org/jira/browse/METRON-942 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > If and after METRON-777 lands, support will be required for installing parser > extensions into the system, as well as uninstalling them. > The first step in this is to have support in metron-rest for this. > The rest interface should support installing a parser extension from the > assembly tar.gz produced by the archetype. > The result of the installation should be: > * the bundle is installed in the alternate extension library location in hdfs > * the patterns for all parsers in the extension should be deployed to hdfs > * the configurations for each pattern should be installed into ZK ( parser, > indexing, enrichment ) > * an extension configuration, that includes the parsers it produces, and the > default configurations ( the above and elasticsearch etc ) should be entered > into ZK > The reading of these configurations should also be supported. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-942) REST Support for Parser Extensions
[ https://issues.apache.org/jira/browse/METRON-942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119942#comment-16119942 ] ASF GitHub Bot commented on METRON-942: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/580 Maintaining METRON-947 is too much of a pain. I have merged it into this PR. > REST Support for Parser Extensions > -- > > Key: METRON-942 > URL: https://issues.apache.org/jira/browse/METRON-942 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > If and after METRON-777 lands, support will be required for installing parser > extensions into the system, as well as uninstalling them. > The first step in this is to have support in metron-rest for this. > The rest interface should support installing a parser extension from the > assembly tar.gz produced by the archetype. > The result of the installation should be: > * the bundle is installed in the alternate extension library location in hdfs > * the patterns for all parsers in the extension should be deployed to hdfs > * the configurations for each pattern should be installed into ZK ( parser, > indexing, enrichment ) > * an extension configuration, that includes the parsers it produces, and the > default configurations ( the above and elasticsearch etc ) should be entered > into ZK > The reading of these configurations should also be supported. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-942) REST Support for Parser Extensions
[
https://issues.apache.org/jira/browse/METRON-942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100751#comment-16100751
]
ASF GitHub Bot commented on METRON-942:
---
Github user BlackHatJohnDoe commented on a diff in the pull request:
https://github.com/apache/metron/pull/580#discussion_r129420751
--- Diff:
metron-maven-archetypes/metron-maven-parser-extension-archetype/src/main/resources/archetype-resources/metron-parser-__parserName__/README.md
---
@@ -0,0 +1,5 @@
+#${parserClassName} Parser
--- End diff --
`s/#${parserClassName}/# ${parserClassName}/`
> REST Support for Parser Extensions
> --
>
> Key: METRON-942
> URL: https://issues.apache.org/jira/browse/METRON-942
> Project: Metron
> Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> If and after METRON-777 lands, support will be required for installing parser
> extensions into the system, as well as uninstalling them.
> The first step in this is to have support in metron-rest for this.
> The rest interface should support installing a parser extension from the
> assembly tar.gz produced by the archetype.
> The result of the installation should be:
> * the bundle is installed in the alternate extension library location in hdfs
> * the patterns for all parsers in the extension should be deployed to hdfs
> * the configurations for each pattern should be installed into ZK ( parser,
> indexing, enrichment )
> * an extension configuration, that includes the parsers it produces, and the
> default configurations ( the above and elasticsearch etc ) should be entered
> into ZK
> The reading of these configurations should also be supported.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-942) REST Support for Parser Extensions
[ https://issues.apache.org/jira/browse/METRON-942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068376#comment-16068376 ] ASF GitHub Bot commented on METRON-942: --- Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/580 > REST Support for Parser Extensions > -- > > Key: METRON-942 > URL: https://issues.apache.org/jira/browse/METRON-942 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > If and after METRON-777 lands, support will be required for installing parser > extensions into the system, as well as uninstalling them. > The first step in this is to have support in metron-rest for this. > The rest interface should support installing a parser extension from the > assembly tar.gz produced by the archetype. > The result of the installation should be: > * the bundle is installed in the alternate extension library location in hdfs > * the patterns for all parsers in the extension should be deployed to hdfs > * the configurations for each pattern should be installed into ZK ( parser, > indexing, enrichment ) > * an extension configuration, that includes the parsers it produces, and the > default configurations ( the above and elasticsearch etc ) should be entered > into ZK > The reading of these configurations should also be supported. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-942) REST Support for Parser Extensions
[
https://issues.apache.org/jira/browse/METRON-942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068377#comment-16068377
]
ASF GitHub Bot commented on METRON-942:
---
GitHub user ottobackwards reopened a pull request:
https://github.com/apache/metron/pull/580
METRON-942 [NO MERGE UNTIL METRON-777] Rest api and configuration for
Metron parser extensions
## Contributor Comments
METRON-777 introduces metron extensions and metron parser extensions. The
existing parsers have been moved to be packaged and loaded as extensions.
A maven archetype for creating parser extensions has also been created.
What is missing however is the ability to install a parser extension
created outside the metron tree by the archetype into a metron system.
This PR introduces a REST api to accomplish this.
Along with that, this PR introduces a parser extension configuration type,
for zookeeper that serves as a registry for installed extensions
and their default configurations. The idea being that in a follow on
effort, the management UI will create a new 'instance' of a parser type
through cloning, editing and deploying these configurations. The included
configurations are indexing, enrichment, parser, and elasticsearch.
There may be multiple of these configurations, as each parser extension
library can contain more than one parser.
NOTE: the elasticsearch template is NOT deployed to elasticsearch at this
time. Any parser created will still need a separate step for this.
One outstanding question for feedback -> The configuration tracks the
parsers created out of the extension, initially. In the future this list should
have new parsers added to it when they are cloned. This would mean we
would want new rest endpoints to edit the configuration ( most likely api to
add parser to config {name} ) added
This PR is built on top of METRON-777, and should not be merged until
METRON-777 is in, and this branched is rebased on to master.
### USAGE
For usage, I'll cover creating a parser and deploying it... and deleting it
using the rest api.
## Preparation
In order to build the parser, we need to build and install the maven plugin
to build the bundles, as well as the maven archetype itself.
At this time, since they are not published to apache maven, this will be a
requirement.
- Build the bundle plugin
```
cd bundles-maven-plugin && mvn -q install && cd ..
```
- Build the archetype
```
cd metron-maven-archetypes/metron-maven-parser-extension-archetype
mvn install
cd ../..
```
## Create and build a new parser extension
- Make a directory and cd into it
- Create from archetype
```
mvn archetype:generate -DarchetypeCatalog=local
```
- choose the org.apache.metron:metron-maven-parser-extension-archetype
(Apache Maven Parser Extension Archetype for Metron) option
- Fill out the information (information on the parameters is in the
README for the archetype)
- NOTE: at this time, the version must match the metronVersion. Maven
is using version in the parent pom chain, and messing up the plugin dependency
- cd into the created directory and build the parser extension
```
mvn package
```
This will result in the tar.gz being created inside the
{NAME}-parser-assembly project/target directory
## start full_dev
- run vagrant up from metron-deployment/vagrant/full_dev_platform
## add a slot to storm
- log into ambari on http://node1:8080
- add a new slot to the storm config
## Open swagger
- log into ambari on http://node1:8080
- go into the metron service | quicklinks
- start swagger with user | password
## Install your parser
In swagger, using the parser-extension-controller
- drop down the POST method
- use the file selector to select the created .tar.gz
- execute
The extension should now be installed:
- you should be able to use the other GET methods to review the generated
configuration.
- the configuration will have the extensionID to use for the other {name}
operations
- you should be able to view the installed files in
hdfs/apps/metron/patterns/{parsername}, hdfs/apps/metron/extensions_alt_lib/
## Start the parser using rest
Still in swagger
- Use the Kafka Controller and the Storm Controller to create a
{parserName} topic, and start a storm job for {parserName}
- In the storm UI you should see the storm instance for the parser
You should be able to verify using the rest api and the various controllers
that the configurations are there and the topics exist and the storm jobs as
well
## Uninstall your parser
In swagger, using the parser-extension-controller
- drop down the DELETE
[jira] [Commented] (METRON-942) REST Support for Parser Extensions
[ https://issues.apache.org/jira/browse/METRON-942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068375#comment-16068375 ] ASF GitHub Bot commented on METRON-942: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/580 TRAVIS! > REST Support for Parser Extensions > -- > > Key: METRON-942 > URL: https://issues.apache.org/jira/browse/METRON-942 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > If and after METRON-777 lands, support will be required for installing parser > extensions into the system, as well as uninstalling them. > The first step in this is to have support in metron-rest for this. > The rest interface should support installing a parser extension from the > assembly tar.gz produced by the archetype. > The result of the installation should be: > * the bundle is installed in the alternate extension library location in hdfs > * the patterns for all parsers in the extension should be deployed to hdfs > * the configurations for each pattern should be installed into ZK ( parser, > indexing, enrichment ) > * an extension configuration, that includes the parsers it produces, and the > default configurations ( the above and elasticsearch etc ) should be entered > into ZK > The reading of these configurations should also be supported. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-942) REST Support for Parser Extensions
[ https://issues.apache.org/jira/browse/METRON-942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042152#comment-16042152 ] ASF GitHub Bot commented on METRON-942: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/580 Another question here - any parser started from the METRON-UI will not start the next time the service starts. This problem exists here as well, and should prob. be another jira > REST Support for Parser Extensions > -- > > Key: METRON-942 > URL: https://issues.apache.org/jira/browse/METRON-942 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > If and after METRON-777 lands, support will be required for installing parser > extensions into the system, as well as uninstalling them. > The first step in this is to have support in metron-rest for this. > The rest interface should support installing a parser extension from the > assembly tar.gz produced by the archetype. > The result of the installation should be: > * the bundle is installed in the alternate extension library location in hdfs > * the patterns for all parsers in the extension should be deployed to hdfs > * the configurations for each pattern should be installed into ZK ( parser, > indexing, enrichment ) > * an extension configuration, that includes the parsers it produces, and the > default configurations ( the above and elasticsearch etc ) should be entered > into ZK > The reading of these configurations should also be supported. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-942) REST Support for Parser Extensions
[ https://issues.apache.org/jira/browse/METRON-942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16002731#comment-16002731 ] Otto Fowler commented on METRON-942: Should not be merged until METRON-777 is merged > REST Support for Parser Extensions > -- > > Key: METRON-942 > URL: https://issues.apache.org/jira/browse/METRON-942 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > If and after METRON-777 lands, support will be required for installing parser > extensions into the system, as well as uninstalling them. > The first step in this is to have support in metron-rest for this. > The rest interface should support installing a parser extension from the > assembly tar.gz produced by the archetype. > The result of the installation should be: > * the bundle is installed in the alternate extension library location in hdfs > * the patterns for all parsers in the extension should be deployed to hdfs > * the configurations for each pattern should be installed into ZK ( parser, > indexing, enrichment ) > * an extension configuration, that includes the parsers it produces, and the > default configurations ( the above and elasticsearch etc ) should be entered > into ZK > The reading of these configurations should also be supported. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
