Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
closing this in preparation for feature branch
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user merrimanr commented on the issue:
https://github.com/apache/metron/pull/530
There are some consequences of not having the bundle classes available in
rest. The list of available classes will only contain metron-parsers homed
parser types so those are the only type of
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
And also, the PR for fixing grok needs to go in to do the next step there
as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
> NOTE On current configure UI limitations
Because I was already tasked with making this PR smaller ( don't laugh ), I
left the Configuration UI and REST not integrated with Bundles.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
OK @merrimanr , @mmiklavc
I have resolved the issues for working with grok from the rest/ui.
[PR INTO 777 for fixing
GROK](https://github.com/ottobackwards/metron/pull/6)
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
@ottobackwards , looking very good. Couple more comments above
(unfortunately the system decided they are on "outdated" code, so you'll have
to press the "Show outdated" buttons), then you
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
I'm seeing this error spinning up topologies in full-dev.
```
1:50.278 o.a.s.util [ERROR] Async loop died!
java.lang.RuntimeException: Grok parser Error: Grok parser unable to
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
@ottobackwards , many good improvements here. A few comments on the
singleton idiom, and a couple other details.
---
If your project is set up for it, you can reply to this email and have
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
@mattf-horton I really like how this is evolving. One thing I have been
thinking of since adding the BundleSystem interface ( which should be the main
external interface ) is that I would
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
*[bundles-lib/src/main/java/org/apache/metron/bundles/BundleThreadContextClassLoader.java,
line 43 at
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
Thanks Matt. I will go through and reply later tonight.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
I will also post the review issues that I do not think have been touched on
yet
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
Okay, @ottobackwards , after all this I really wanted to just say "it's
good" :-) but I did have a few more concerns, especially with
ExtensionManager.java.
I want to emphasize that this
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
in fact, you will see more in the management ui, since the extension
parsers all have default configurations and will be present :)
---
If your project is set up for it, you can reply to this
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
The management ui works with zookeeper. You will be able to work with it.
Adding support for installing extensions through the UI is a follow on, but I
am not sure that I can do it
---
If
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
We discussed this higher up in the pr and decided not to
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
Does it make sense to create a feature branch for these PR's?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
@mmiklavc keep in mind that the proper scope for the document is building
new parsers that are 'part' of metron for METRON-777.
METRON-942 has the capability to actually install and
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
Hey @ottobackwards. I see no benefit to getting this committed prior to or
separate from shoring up the docs. I'm going to work on something to get the
ball rolling as I work through some use
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
Test
---
*Comments from
[Reviewable](https://reviewable.io:443/reviews/apache/metron/530#-:-Kr8-4J5YPoUugdlItUi:bb74njr)*
---
If your
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
Yes, and there were not a lot in NAR to start with
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
Sure, no worries. And I didn't intend to imply that testing was
inadequate, just suggesting another for completeness. Can't have too many
tests :-)
---
If your project is set up for it, you
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
Yes, I am sorry, I just wanted to point out another test, outside the area
that you are currently looking. I did not mean to imply that it negated the
need for 1099.
Although, if we
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
@ottobackwards , re `metron-parser-bundle-tests`, very good to have that
test. But it only loads one test bundle, right? so still would be good to
implement METRON-1099. Emphasizing that
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
@mmiklavc The documentation also changes with METRON-942, as that includes
the REST installation steps. If we can get these two PR's through, then follow
on with improved docs, it may make
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
@mmiklavc can the new document be a follow on? The jira with your writeup
would be a good one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
@mmiklavc "useful and overwhelming at the same time". If I had a nickelâ¢.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
@mattf-horton I don't know if you have seen it, but there is an integration
test that tests parser but ensures that the bundle is loaded and not in the
default classloader...
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
@ottobackwards , the \@VisibleForTesting annotation comes from:
> import com.google.common.annotations.VisibleForTesting;
which I believe comes from
```xml
18.0
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
I've opened METRON-1099 for integration tests regarding the two items I'm
not sure from code inspection will work right. But I'm not making this review
dependent on them because they are a
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
This looks much better now. **Note, I also changed the archetype packaging
version to 3.0.1 as well since this is the latest.**
```
{11:39}/tmp â mvn archetype:generate
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
I think I've figured out what's going on with the plugin versions. I was
under the impression that Maven updated its local metadata on at least a daily
basis. It would appear that this is not true.
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
@ottobackwards , this is great work! Sorry the review's so long, and it
still isn't quite done. But some good news: I've fully reviewed
- bundles-maven-plugin
-
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
Running the generate command in CentOS results in defaulting to plugin
version 3.0.1 when run from a directory without a pom. In Mac, I'm getting 2.4
for some reason. I haven't found any settings
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
I just noticed that you had addressed my findings around the plugin version
as well. So I think going back to 3.0.1 is ok for now. We do need to figure out
what is going on with Mac OS before
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
I have changed it back to 3.0.1.
I have also resolved the issue that required the metron version and
archetype version to match in order to build.
---
If your project is set up for it,
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
@mmiklavc does that mean we can go back to 3.0.1? Or should?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
Ok, I decided to try yet another combo here. I set the plugin version back
to 3.0.1 and re-installed the archetype. Now I get a new file
`~/.m2/repository/archetype-catalog.xml` with the archetype
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
So another wrinkle here - when running the archetype plugin from a
non-project (separate, clean) directory, it's going to default to 3.0.1. I'm
trying this up in full dev right now and running into
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
ok - so mvn archetype:generate uses the newer version of the plugin if run
in an empty directory.
If run in a directory where there is a pom with plugin setting for 2.4 and
it will honor
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
hey @mmiklavc . Looking back at my logs above, no matter that the
maven-archetype-plugin was 2.4 in the pom at the time, it shows 3.0.1 in the
output. Can you check what version it shows in
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
I also may have a fix for the missing catalog file issue ( or a way to run
the command differently )
---
If your project is set up for it, you can reply to this email and have your
reply
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
Ok, I tried with a new M2 directory and the can't find local archetype
issue returned.
maven-archetype-plugin
3.0.1
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
I'm going to try with a new .m2 dir
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
I hate to ask but are you sure you have the latest code?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
Hrm, I completely blew away both archetype catalogs and re-installed our
archetype again. It shows up with the correct version. However, the parser
folders are still not being created correctly. As
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
```bash
ââ[ottofowler@Winterfell] - [~/tmp/HelloParser] - [Thu Aug 03, 19:12]
ââ[$]> cat ~/.m2/repository/archetype-catalog.xml
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
I left the Parser off of the HelloParser input, sorry
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
```
cat ~/.m2/repository/archetype-catalog.xml
http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0
http://maven.apache.org/xsd/archetype-catalog-1.0.0.xsd;
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
I DO think that the archetype's input variables could use refactoring. For
example, I force the artifact name, and maybe I should not.
I do not know why the bundle plugin is looking
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
```bash
ââ[ottofowler@Winterfell] -
[~/.m2/repository/org/apache/metron/metron-maven-parser-extension-archetype/0.4.1]
- [Thu Aug 03, 18:42]
ââ[$]> ll
total 80
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
```bash
ââ[ottofowler@Winterfell] - [~/tmp] - [Thu Aug 03, 18:36]
ââ[$]> mvn archetype:generate -DarchetypeCatalog=local
[INFO] Scanning for projects...
[INFO]
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
I'm not done the build, so I'll try this again, but this is what I get: (
Note, I don't know how you are getting the packageInPathFormat option ):
ââ[ottofowler@Winterfell] -
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
WRT: the archetype directories: I don't see that ( but I'm going to
re-build and try again ).
I don't use the same options you are using though.
Importantly -> the parser version
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
Ok, I see what that's doing now re: metron-parsers. As I was glancing over
the project dirs my next question was going to be about what is in
metron-parsers vs the extensions, but I follow. I do
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
Will review your other comments shortly. Some additional questions:
I ran the archetype to create a new system parser:
```
[INFO] Parameter: groupId, Value: com.michaelmiklavcic
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
Ok - I think you are running find from /usr/metron/$V/
So - to explain the other config/zookeeper
So let's pretend there are two things here:
1. metron-parsers
2.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
thanks @mmiklavc. I don't think I've stated this in the actual PR
As a 3rd party ( or whatever you want to call it ) developer, who only
makes Parser or possibly other extensions
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
I see that for parser extensions, the enrichment and indexing
configurations have also been moved into this tree. This structure conflates
parsers, enrichments, and indexing. Beyond that, I also
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
Oh profiler integration test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
@mmiklavc
1. I don't think we want to, or have to track it as upstream. But we do
want to keep track of bug fixes and improvements to areas where we have not
deviated. That is my
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
I'm back to looking through this again. A couple questions:
1) I was under the impression that our local copy of the nar functionality
would be a hard fork that would naturally deviate from the
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
"Based on feedback from reviews and discussion with mattf, this commitâ¦
⦠represent a major refactoring for the bundle functionality.
While trying to resolve the ugliness bought
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
and thanks for the file map of the test area.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
@ottobackwards , I agree it isn't critical to update now, and would slow
down this PR even more. Do you mind opening a tracking jira to consider an
update in the future? (assuming they
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
@mattf-horton I have reviewed the changes between what I have and 1.3.0,
and while there are some one or two slight differences, there is nothing there
that functionally improves or applies to
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
```bash
.
âââ java
â  âââ org
â  âââ apache
â  âââ metron
â  âââ bundles
â Â
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
No, there is no problem with you referring back to nifi @simonellistonball
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user simonellistonball commented on the issue:
https://github.com/apache/metron/pull/530
@ottobackwards I didn't steal _that_ much of it. Still it shouldn't cause a
problem, as long as we get all the updates since this PR started ported in.
That one has moved a bit from things
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
@ottobackwards , can you please give me a similar file mapping for the
"test" side of bundles-lib, ie bundles-lib/src/test/java/org/apache/metron/ vs
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
WRT: Required Bundle dependency for BundleProvidedDependenciesMojo.
That mojo is not run automatically, either in metron or in the nifi case.
It is *explicitly* run to determine the
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
I'm actually going to debug this and verify about the dependency
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
> My question was, since it throws an exception if no Bundle dependency is
stated, if I have a Bundle that is not actually dependent on any other bundles,
what should I declare it is dependent
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
```code
-
To have a single lib, some things that were in other nifi libraries had to
be relocated.
./bundles:
ExtensionManager.java-> MODIFIED
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
Thanks for the feedback @mattf-horton, I will go through it all. About
tracking changes... I will check and see what there is to pick up. Obviously
this was much more current when the PR was
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
Please remove instances of "incubator" or "incubating" in bundles-lib/,
bundles-maven-plugin/, and metron-platform/metron-extensions :
```
bundles-maven-plugin/pom.xml:
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
I'm trying to compare bundle-lib with nifi-nar-utils, and blocked because
the code trees under
* bundles-lib/src/main/java/org/apache/metron/bundles and
*
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
Hi @ottobackwards , the PR description list of files changed and reasons is
_extremely_ helpful, indeed necessary. But the github PR GUI isn't really
tuned for handling 400 files even when --
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
@mattf-horton I believe
https://github.com/apache/nifi/commit/d90cf846b9dcd49acdeac7bb5fcd13ee80dcdba6#diff-83e1afb34470ca47809f82aa1caf2138
is the commit from nifi-nar-utils
---
If your
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
@mattf-horton branch is up to date as of this morning
https://github.com/apache/metron/commit/badc6cf9739a31800abee1cfccbcf9930b130fa7
---
If your project is set up for it, you can reply
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
https://github.com/apache/metron/pull/530/commits/0d9a43a9393a20309845c668a206005f45ea4617
for our master as of this moment @mattf-horton
---
If your project is set up for it, you
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
@mattf-horton In the PR description there is a list of files changed and
the reason for them, does that help?
I am going to merge master and resolve conflicts this morning. I have
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
@ottobackwards , could you please state the latest SHA1 on master to which
you've merged?
I ask because I'm going to need to bring up a full dev image, and do `git
diff`s against the
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
I'm working on this review. It's gonna take a couple days.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
Have you been able to try this? Is there any other feedback?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
Archetype plugin version is the only change
```
http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
@mmiklavc can you comment with what your pom looks like now?
Why does 3.0 work on my machine though.
---
If your project is set up for it, you can reply to this email and have your
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
I'm ok with that, I would feel better if we got a tie breaking opinion
though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
Ok, I was getting ready to bail on this for the day but couldn't stop. I
tried one more thing - setting the maven-archetype-plugin version to **2.4**
and it works as expected. Something must be
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
@ottobackwards removing the extra project element allows me to build the
project now. However, I'm still unable to use the archetype. It neither shows
up in the local catalog listing nor allows me
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
I have added deployment readme files to document where things get deployed
to.
This if for the built in system parser extensions. The 3rd party
extensions are deployed by the rest api in
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
OK, IWOMM.
I'm on irc if you want to jump on
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/530
Looking into it now, but building locally I see this:
```
$ mvn clean install -DskipTests -T 2C
...
[INFO] elasticsearch-shaded ... SKIPPED
[INFO]
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
I have added more documentation and I'm working on documenting where things
are deployed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
Thank you @mattf-horton, I was able to fix the links.
So, @mmiklavc et. al., can we frame our documentation discussion around
filling out and improving these documents?
---
If
Github user mattf-horton commented on the issue:
https://github.com/apache/metron/pull/530
@ottobackwards , in general (a) make it work in github, and (b) make sure
the human-readable link reference text is the same as the human-readable anchor
text. That way the site-book generator
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
If we want to come up with a different name for the assembly ( but it will
still be tgz ) I would be open to that. I did not want to go that far without
review
---
If your project is set up
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
I will try to come up with a diagram about the assembly composition with a
dictionary
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
@merrimanr , I am not sure how to fix metron-docker to work with this. The
parsers are deployed to hdfs now as extensions, there are different things in
zookeeper, the ambari integration etc.
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/530
I have moved metron-extensions to live in metron-platform
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
1 - 100 of 139 matches
Mail list logo
