Re: [Discuss] SIDELOADING PARSERS: Parsers as components

[Otto Fowler]

The RPM’s are only a requirement for installation by ambari.

We could drop in Nars to the system location ( hdfs in the end ).
I imagine us doing that through the UI.

Nar’s *are* automatically unpacked at execution time.

Nar’s are unpackaged into a ‘working’ area ( if required ), that working
area is the class path that is
loaded.




On March 14, 2017 at 09:57:14, Kyle Richardson (kylerichards...@gmail.com)
wrote:

Solid work, Otto. I'm excited to see us move in this direction. It's an
important step to making Metron a more user friendly platform.

I agree with Matt. I think a PR for this piece is needed sooner rather than
later so you don't continually have to rework it as we make more commits
into master.

One question/concern/future thought, is there anyway we can get away from
needing the RPMs for these extensible components? Ideally, I'd like to be
able to dump my parser package file (jar, nar, etc) into a directory and
have it automagically unpacked.

-Kyle

On Mon, Mar 13, 2017 at 10:10 AM, Otto Fowler 
wrote:

> James,
> Can you clarify your concerns about backward compatibility?
>
>
>
> On March 12, 2017 at 23:48:17, James Sirota (jsir...@apache.org) wrote:
>
> As long as this doesn't break backwards compatibility I think I am ok
with
> this approach. I think this is a great idea. We would probably need to
> follow up with an Ambari view that can allow you to list and
> deploy/upgrade/delete various parsers
>
> 10.03.2017, 15:16, "Casey Stella" :
> > Ok, so, after some thought about this, I am in agreement over Nar. I do
> > want to make sure that on the roadmap we retrofit stellar to accept Nar
> > plugins and build an archetype for it. We should have a single strategy
> > for plugins. NOt saying it has to be part of the same PR, but it needs
to
> > be associated and a follow-on task IMO.
> >
> > On Fri, Mar 10, 2017 at 4:06 PM, Otto Fowler 
> > wrote:
> >
> >> Also a Nar can depend on ‘one’ other nar, which is interesting
> >>
> >> On March 10, 2017 at 16:02:18, Otto Fowler (ottobackwa...@gmail.com)
> >> wrote:
> >>
> >> The isolation is just a ‘extra’ in the parser case.
> >>
> >> The parts of Nar that *are* more pertinent:
> >>
> >> * supporting a deployment artifact with just a jar, or a tar.gz with a
> jar
> >> and jar dependencies in it
> >> * taking a ‘package’ and deploying it for loading ( which will upgrade
> the
> >> deployed part if it is newer )
> >> * setting up the classloader hierarchy between the ‘system’ and
> provided
> >> things, and the dependencies of the individual plugin
> >>
> >> On March 10, 2017 at 15:56:08, Casey Stella (ceste...@gmail.com)
> wrote:
> >>
> >> Why would we need classpath isolation here in the case of the parser?
> >>
> >> On Fri, Mar 10, 2017 at 3:55 PM, Otto Fowler 
> >> wrote:
> >>
> >>> I *would* use the classloader part, extending it with VFS.
> >>>
> >>> On March 10, 2017 at 15:53:05, Casey Stella (ceste...@gmail.com)
> wrote:
> >>>
> >>> I'm a bit worried about copying and pasting from the NiFi project
> their
> >>> nar infrastructure. That seems..unclean to me and since we're not
> using
> >>> the classloader part of nar for this, does it make more sense to just
> use
> >>> jar?
> >>>
> >>> On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler  >
> >>> wrote:
> >>>
>  Compared to how much time vagrant up takes now, you won’t even
notice
> it
>  ;)
> 
>  That is definitely an option. I guess what I want to work out is if
> we
>  are going to want to
>  go to NAR, why not just go to NAR.
> 
>  In the end, the customer for this - like Jon Zeolla, isn’t going to
> care
>  about the intermediate step,
>  he wants the archetype that builds the ‘metron parser plugin’.
> 
>  Which is why I hesitate to put out an archetype that is going to
>  obsolete so soon.
> 
>  Does that make sense?
> 
>  On March 10, 2017 at 14:50:55, Casey Stella (ceste...@gmail.com)
> wrote:
> 
>  I'm a little concerned about this increasing the size and length of
> the
>  build due to the repeated shading. Should we figure out a way to
> deploy
>  jars with provided dependencies on metron-parser-common as suggested
> in
>  the
>  previous JIRAs first?
> 
>  On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley 
>  wrote:
> 
>  > It sounds like:
>  > - This is a self-contained chunk of work, that can be tested,
> reviewed,
>  > and committed on its own, then the other ideas you propose can
> follow
>  it.
>  > - It crosses a lot of lines, and restructures a lot of code, so
> will
>  “rot”
>  > fairly quickly as other people make commits, so if possible you
> should
>  get
>  > a PR out there and we should work through it as soon as possible.
>  > Are those both true?
>  >
>  > How do other people 

Re: [Discuss] SIDELOADING PARSERS: Parsers as components

[Kyle Richardson]

Gotcha. Makes sense.

Unrelated, but something I've had in the back of my head for a while... I
think we should try to loosely define what roles would be using Ambari vs
Metron UI vs CLI. For example, I'm thinking there may be a difference
between the cluster admin and the Metron admin in an organization. With
that, I think the idea of using the Metron UI to load parser packages makes
a lot of sense.

On Tue, Mar 14, 2017 at 10:39 AM, Otto Fowler 
wrote:

> The RPM’s are only a requirement for installation by ambari.
>
> We could drop in Nars to the system location ( hdfs in the end ).
> I imagine us doing that through the UI.
>
> Nar’s *are* automatically unpacked at execution time.
>
> Nar’s are unpackaged into a ‘working’ area ( if required ), that working
> area is the class path that is
> loaded.
>
>
>
>
> On March 14, 2017 at 09:57:14, Kyle Richardson (kylerichards...@gmail.com)
> wrote:
>
> Solid work, Otto. I'm excited to see us move in this direction. It's an
> important step to making Metron a more user friendly platform.
>
> I agree with Matt. I think a PR for this piece is needed sooner rather
> than
> later so you don't continually have to rework it as we make more commits
> into master.
>
> One question/concern/future thought, is there anyway we can get away from
> needing the RPMs for these extensible components? Ideally, I'd like to be
> able to dump my parser package file (jar, nar, etc) into a directory and
> have it automagically unpacked.
>
> -Kyle
>
> On Mon, Mar 13, 2017 at 10:10 AM, Otto Fowler 
> wrote:
>
> > James,
> > Can you clarify your concerns about backward compatibility?
> >
> >
> >
> > On March 12, 2017 at 23:48:17, James Sirota (jsir...@apache.org) wrote:
> >
> > As long as this doesn't break backwards compatibility I think I am ok
> with
> > this approach. I think this is a great idea. We would probably need to
> > follow up with an Ambari view that can allow you to list and
> > deploy/upgrade/delete various parsers
> >
> > 10.03.2017, 15:16, "Casey Stella" :
> > > Ok, so, after some thought about this, I am in agreement over Nar. I
> do
> > > want to make sure that on the roadmap we retrofit stellar to accept
> Nar
> > > plugins and build an archetype for it. We should have a single
> strategy
> > > for plugins. NOt saying it has to be part of the same PR, but it needs
> to
> > > be associated and a follow-on task IMO.
> > >
> > > On Fri, Mar 10, 2017 at 4:06 PM, Otto Fowler 
>
> > > wrote:
> > >
> > >> Also a Nar can depend on ‘one’ other nar, which is interesting
> > >>
> > >> On March 10, 2017 at 16:02:18, Otto Fowler (ottobackwa...@gmail.com)
> > >> wrote:
> > >>
> > >> The isolation is just a ‘extra’ in the parser case.
> > >>
> > >> The parts of Nar that *are* more pertinent:
> > >>
> > >> * supporting a deployment artifact with just a jar, or a tar.gz with
> a
> > jar
> > >> and jar dependencies in it
> > >> * taking a ‘package’ and deploying it for loading ( which will
> upgrade
> > the
> > >> deployed part if it is newer )
> > >> * setting up the classloader hierarchy between the ‘system’ and
> > provided
> > >> things, and the dependencies of the individual plugin
> > >>
> > >> On March 10, 2017 at 15:56:08, Casey Stella (ceste...@gmail.com)
> > wrote:
> > >>
> > >> Why would we need classpath isolation here in the case of the parser?
> > >>
> > >> On Fri, Mar 10, 2017 at 3:55 PM, Otto Fowler 
>
> > >> wrote:
> > >>
> > >>> I *would* use the classloader part, extending it with VFS.
> > >>>
> > >>> On March 10, 2017 at 15:53:05, Casey Stella (ceste...@gmail.com)
> > wrote:
> > >>>
> > >>> I'm a bit worried about copying and pasting from the NiFi project
> > their
> > >>> nar infrastructure. That seems..unclean to me and since we're not
> > using
> > >>> the classloader part of nar for this, does it make more sense to
> just
> > use
> > >>> jar?
> > >>>
> > >>> On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler <
> ottobackwa...@gmail.com
> > >
> > >>> wrote:
> > >>>
> >  Compared to how much time vagrant up takes now, you won’t even
> notice
> > it
> >  ;)
> > 
> >  That is definitely an option. I guess what I want to work out is if
> > we
> >  are going to want to
> >  go to NAR, why not just go to NAR.
> > 
> >  In the end, the customer for this - like Jon Zeolla, isn’t going to
> > care
> >  about the intermediate step,
> >  he wants the archetype that builds the ‘metron parser plugin’.
> > 
> >  Which is why I hesitate to put out an archetype that is going to
> >  obsolete so soon.
> > 
> >  Does that make sense?
> > 
> >  On March 10, 2017 at 14:50:55, Casey Stella (ceste...@gmail.com)
> > wrote:
> > 
> >  I'm a little concerned about this increasing the size and length of
> > the
> >  build due to the repeated shading. Should we figure out a way to
> > deploy
> > 

Re: [Discuss] SIDELOADING PARSERS: Parsers as components

[Kyle Richardson]

Solid work, Otto. I'm excited to see us move in this direction. It's an
important step to making Metron a more user friendly platform.

I agree with Matt. I think a PR for this piece is needed sooner rather than
later so you don't continually have to rework it as we make more commits
into master.

One question/concern/future thought, is there anyway we can get away from
needing the RPMs for these extensible components? Ideally, I'd like to be
able to dump my parser package file (jar, nar, etc) into a directory and
have it automagically unpacked.

-Kyle

On Mon, Mar 13, 2017 at 10:10 AM, Otto Fowler 
wrote:

> James,
> Can you clarify your concerns about backward compatibility?
>
>
>
> On March 12, 2017 at 23:48:17, James Sirota (jsir...@apache.org) wrote:
>
> As long as this doesn't break backwards compatibility I think I am ok with
> this approach. I think this is a great idea. We would probably need to
> follow up with an Ambari view that can allow you to list and
> deploy/upgrade/delete various parsers
>
> 10.03.2017, 15:16, "Casey Stella" :
> > Ok, so, after some thought about this, I am in agreement over Nar. I do
> > want to make sure that on the roadmap we retrofit stellar to accept Nar
> > plugins and build an archetype for it. We should have a single strategy
> > for plugins. NOt saying it has to be part of the same PR, but it needs to
> > be associated and a follow-on task IMO.
> >
> > On Fri, Mar 10, 2017 at 4:06 PM, Otto Fowler 
> > wrote:
> >
> >>  Also a Nar can depend on ‘one’ other nar, which is interesting
> >>
> >>  On March 10, 2017 at 16:02:18, Otto Fowler (ottobackwa...@gmail.com)
> >>  wrote:
> >>
> >>  The isolation is just a ‘extra’ in the parser case.
> >>
> >>  The parts of Nar that *are* more pertinent:
> >>
> >>  * supporting a deployment artifact with just a jar, or a tar.gz with a
> jar
> >>  and jar dependencies in it
> >>  * taking a ‘package’ and deploying it for loading ( which will upgrade
> the
> >>  deployed part if it is newer )
> >>  * setting up the classloader hierarchy between the ‘system’ and
> provided
> >>  things, and the dependencies of the individual plugin
> >>
> >>  On March 10, 2017 at 15:56:08, Casey Stella (ceste...@gmail.com)
> wrote:
> >>
> >>  Why would we need classpath isolation here in the case of the parser?
> >>
> >>  On Fri, Mar 10, 2017 at 3:55 PM, Otto Fowler 
> >>  wrote:
> >>
> >>>  I *would* use the classloader part, extending it with VFS.
> >>>
> >>>  On March 10, 2017 at 15:53:05, Casey Stella (ceste...@gmail.com)
> wrote:
> >>>
> >>>  I'm a bit worried about copying and pasting from the NiFi project
> their
> >>>  nar infrastructure. That seems..unclean to me and since we're not
> using
> >>>  the classloader part of nar for this, does it make more sense to just
> use
> >>>  jar?
> >>>
> >>>  On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler  >
> >>>  wrote:
> >>>
>   Compared to how much time vagrant up takes now, you won’t even notice
> it
>   ;)
> 
>   That is definitely an option. I guess what I want to work out is if
> we
>   are going to want to
>   go to NAR, why not just go to NAR.
> 
>   In the end, the customer for this - like Jon Zeolla, isn’t going to
> care
>   about the intermediate step,
>   he wants the archetype that builds the ‘metron parser plugin’.
> 
>   Which is why I hesitate to put out an archetype that is going to
>   obsolete so soon.
> 
>   Does that make sense?
> 
>   On March 10, 2017 at 14:50:55, Casey Stella (ceste...@gmail.com)
> wrote:
> 
>   I'm a little concerned about this increasing the size and length of
> the
>   build due to the repeated shading. Should we figure out a way to
> deploy
>   jars with provided dependencies on metron-parser-common as suggested
> in
>   the
>   previous JIRAs first?
> 
>   On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley 
>   wrote:
> 
>   > It sounds like:
>   > - This is a self-contained chunk of work, that can be tested,
> reviewed,
>   > and committed on its own, then the other ideas you propose can
> follow
>   it.
>   > - It crosses a lot of lines, and restructures a lot of code, so
> will
>   “rot”
>   > fairly quickly as other people make commits, so if possible you
> should
>   get
>   > a PR out there and we should work through it as soon as possible.
>   > Are those both true?
>   >
>   > How do other people feel about grouping a given sensor’s parser,
>   enricher,
>   > indexing logic all together? It seems to have multiple advantages
> are
>   > there also disadvantages?
>   >
>   > On 3/10/17, 6:31 AM, "Otto Fowler" 
> wrote:
>   >
>   > As previously discussed here, I have been working on side loading
> of
>   > 

Re: [Discuss] SIDELOADING PARSERS: Parsers as components

[Otto Fowler]

James,
Can you clarify your concerns about backward compatibility?



On March 12, 2017 at 23:48:17, James Sirota (jsir...@apache.org) wrote:

As long as this doesn't break backwards compatibility I think I am ok with
this approach. I think this is a great idea. We would probably need to
follow up with an Ambari view that can allow you to list and
deploy/upgrade/delete various parsers

10.03.2017, 15:16, "Casey Stella" :
> Ok, so, after some thought about this, I am in agreement over Nar. I do
> want to make sure that on the roadmap we retrofit stellar to accept Nar
> plugins and build an archetype for it. We should have a single strategy
> for plugins. NOt saying it has to be part of the same PR, but it needs to
> be associated and a follow-on task IMO.
>
> On Fri, Mar 10, 2017 at 4:06 PM, Otto Fowler 
> wrote:
>
>>  Also a Nar can depend on ‘one’ other nar, which is interesting
>>
>>  On March 10, 2017 at 16:02:18, Otto Fowler (ottobackwa...@gmail.com)
>>  wrote:
>>
>>  The isolation is just a ‘extra’ in the parser case.
>>
>>  The parts of Nar that *are* more pertinent:
>>
>>  * supporting a deployment artifact with just a jar, or a tar.gz with a
jar
>>  and jar dependencies in it
>>  * taking a ‘package’ and deploying it for loading ( which will upgrade
the
>>  deployed part if it is newer )
>>  * setting up the classloader hierarchy between the ‘system’ and
provided
>>  things, and the dependencies of the individual plugin
>>
>>  On March 10, 2017 at 15:56:08, Casey Stella (ceste...@gmail.com) wrote:
>>
>>  Why would we need classpath isolation here in the case of the parser?
>>
>>  On Fri, Mar 10, 2017 at 3:55 PM, Otto Fowler 
>>  wrote:
>>
>>>  I *would* use the classloader part, extending it with VFS.
>>>
>>>  On March 10, 2017 at 15:53:05, Casey Stella (ceste...@gmail.com)
wrote:
>>>
>>>  I'm a bit worried about copying and pasting from the NiFi project
their
>>>  nar infrastructure. That seems..unclean to me and since we're not
using
>>>  the classloader part of nar for this, does it make more sense to just
use
>>>  jar?
>>>
>>>  On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler 
>>>  wrote:
>>>
  Compared to how much time vagrant up takes now, you won’t even notice
it
  ;)

  That is definitely an option. I guess what I want to work out is if
we
  are going to want to
  go to NAR, why not just go to NAR.

  In the end, the customer for this - like Jon Zeolla, isn’t going to
care
  about the intermediate step,
  he wants the archetype that builds the ‘metron parser plugin’.

  Which is why I hesitate to put out an archetype that is going to
  obsolete so soon.

  Does that make sense?

  On March 10, 2017 at 14:50:55, Casey Stella (ceste...@gmail.com)
wrote:

  I'm a little concerned about this increasing the size and length of
the
  build due to the repeated shading. Should we figure out a way to
deploy
  jars with provided dependencies on metron-parser-common as suggested
in
  the
  previous JIRAs first?

  On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley 
  wrote:

  > It sounds like:
  > - This is a self-contained chunk of work, that can be tested,
reviewed,
  > and committed on its own, then the other ideas you propose can
follow
  it.
  > - It crosses a lot of lines, and restructures a lot of code, so
will
  “rot”
  > fairly quickly as other people make commits, so if possible you
should
  get
  > a PR out there and we should work through it as soon as possible.
  > Are those both true?
  >
  > How do other people feel about grouping a given sensor’s parser,
  enricher,
  > indexing logic all together? It seems to have multiple advantages
are
  > there also disadvantages?
  >
  > On 3/10/17, 6:31 AM, "Otto Fowler"  wrote:
  >
  > As previously discussed here, I have been working on side loading
of
  > parsers. The goals of this work are:
  > * Make it possible of developers to create, maintain and deploy
parsers
  > outside of the Metron code tree and not have to fork
  > * Create maven archetype support for developers of parsers
  > * Introduce a parser ‘lifecycle’ to support multiple instances and
  > configurations, states of being installed, under configuration, and
  > deployed
  > etc.
  >
  > I would like to have some discussion based on where I am after
rebasing
  > onto METRON-671 which revamps deployment to be totally ambari
based.
  >
  >
  > Parsers as components:
  >
  > I have all the parsers broken out into individual
packages/rpms/jars.
  > What I have done is taken metron-parsers and broken it out to:
  >
  > * metron-parsers-common
  > * This has all the base classes and 

Re: [Discuss] SIDELOADING PARSERS: Parsers as components

[James Sirota]

As long as this doesn't break backwards compatibility I think I am ok with this 
approach.  I think this is a great idea.  We would probably need to follow up 
with an Ambari view that can allow you to list and deploy/upgrade/delete 
various parsers

10.03.2017, 15:16, "Casey Stella" :
> Ok, so, after some thought about this, I am in agreement over Nar. I do
> want to make sure that on the roadmap we retrofit stellar to accept Nar
> plugins and build an archetype for it. We should have a single strategy
> for plugins. NOt saying it has to be part of the same PR, but it needs to
> be associated and a follow-on task IMO.
>
> On Fri, Mar 10, 2017 at 4:06 PM, Otto Fowler 
> wrote:
>
>>  Also a Nar can depend on ‘one’ other nar, which is interesting
>>
>>  On March 10, 2017 at 16:02:18, Otto Fowler (ottobackwa...@gmail.com)
>>  wrote:
>>
>>  The isolation is just a ‘extra’ in the parser case.
>>
>>  The parts of Nar that *are* more pertinent:
>>
>>  * supporting a deployment artifact with just a jar, or a tar.gz with a jar
>>  and jar dependencies in it
>>  * taking a ‘package’ and deploying it for loading ( which will upgrade the
>>  deployed part if it is newer )
>>  * setting up the classloader hierarchy between the ‘system’ and provided
>>  things, and the dependencies of the individual plugin
>>
>>  On March 10, 2017 at 15:56:08, Casey Stella (ceste...@gmail.com) wrote:
>>
>>  Why would we need classpath isolation here in the case of the parser?
>>
>>  On Fri, Mar 10, 2017 at 3:55 PM, Otto Fowler 
>>  wrote:
>>
>>>  I *would* use the classloader part, extending it with VFS.
>>>
>>>  On March 10, 2017 at 15:53:05, Casey Stella (ceste...@gmail.com) wrote:
>>>
>>>  I'm a bit worried about copying and pasting from the NiFi project their
>>>  nar infrastructure. That seems..unclean to me and since we're not using
>>>  the classloader part of nar for this, does it make more sense to just use
>>>  jar?
>>>
>>>  On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler 
>>>  wrote:
>>>
  Compared to how much time vagrant up takes now, you won’t even notice it
  ;)

  That is definitely an option. I guess what I want to work out is if we
  are going to want to
  go to NAR, why not just go to NAR.

  In the end, the customer for this - like Jon Zeolla, isn’t going to care
  about the intermediate step,
  he wants the archetype that builds the ‘metron parser plugin’.

  Which is why I hesitate to put out an archetype that is going to
  obsolete so soon.

  Does that make sense?

  On March 10, 2017 at 14:50:55, Casey Stella (ceste...@gmail.com) wrote:

  I'm a little concerned about this increasing the size and length of the
  build due to the repeated shading. Should we figure out a way to deploy
  jars with provided dependencies on metron-parser-common as suggested in
  the
  previous JIRAs first?

  On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley 
  wrote:

  > It sounds like:
  > - This is a self-contained chunk of work, that can be tested, reviewed,
  > and committed on its own, then the other ideas you propose can follow
  it.
  > - It crosses a lot of lines, and restructures a lot of code, so will
  “rot”
  > fairly quickly as other people make commits, so if possible you should
  get
  > a PR out there and we should work through it as soon as possible.
  > Are those both true?
  >
  > How do other people feel about grouping a given sensor’s parser,
  enricher,
  > indexing logic all together? It seems to have multiple advantages are
  > there also disadvantages?
  >
  > On 3/10/17, 6:31 AM, "Otto Fowler"  wrote:
  >
  > As previously discussed here, I have been working on side loading of
  > parsers. The goals of this work are:
  > * Make it possible of developers to create, maintain and deploy parsers
  > outside of the Metron code tree and not have to fork
  > * Create maven archetype support for developers of parsers
  > * Introduce a parser ‘lifecycle’ to support multiple instances and
  > configurations, states of being installed, under configuration, and
  > deployed
  > etc.
  >
  > I would like to have some discussion based on where I am after rebasing
  > onto METRON-671 which revamps deployment to be totally ambari based.
  >
  >
  > Parsers as components:
  >
  > I have all the parsers broken out into individual packages/rpms/jars.
  > What I have done is taken metron-parsers and broken it out to:
  >
  > * metron-parsers-common
  > * This has all the base classes and interfaces, common testing
  > components
  > etc
  > * metron-parser-base
  > * This has the Grok, CSV, and JsonMap parsers and 

Re: [Discuss] SIDELOADING PARSERS: Parsers as components

[Casey Stella]

Ok, so, after some thought about this, I am in agreement over Nar.  I do
want to make sure that on the roadmap we retrofit stellar to accept Nar
plugins and build an archetype for it.  We should have a single strategy
for plugins.  NOt saying it has to be part of the same PR, but it needs to
be associated and a follow-on task IMO.

On Fri, Mar 10, 2017 at 4:06 PM, Otto Fowler 
wrote:

> Also a Nar can depend on ‘one’ other nar, which is interesting
>
>
> On March 10, 2017 at 16:02:18, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> The isolation is just a ‘extra’ in the parser case.
>
> The parts of Nar that *are* more pertinent:
>
> * supporting a deployment artifact with just a jar, or a tar.gz with a jar
> and jar dependencies in it
> * taking a ‘package’ and deploying it for loading ( which will upgrade the
> deployed part if it is newer )
> * setting up the classloader hierarchy between the ‘system’ and provided
> things, and the dependencies of the individual plugin
>
>
>
> On March 10, 2017 at 15:56:08, Casey Stella (ceste...@gmail.com) wrote:
>
> Why would we need classpath isolation here in the case of the parser?
>
> On Fri, Mar 10, 2017 at 3:55 PM, Otto Fowler 
> wrote:
>
>> I *would* use the classloader part, extending it with VFS.
>>
>>
>>
>> On March 10, 2017 at 15:53:05, Casey Stella (ceste...@gmail.com) wrote:
>>
>> I'm a bit worried about copying and pasting from the NiFi project their
>> nar infrastructure.  That seems..unclean to me and since we're not using
>> the classloader part of nar for this, does it make more sense to just use
>> jar?
>>
>> On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler 
>> wrote:
>>
>>> Compared to how much time vagrant up takes now, you won’t even notice it
>>> ;)
>>>
>>> That is definitely an option.  I guess what I want to work out is if we
>>> are going to want to
>>> go to NAR, why not just go to NAR.
>>>
>>> In the end, the customer for this - like Jon Zeolla, isn’t going to care
>>> about the intermediate step,
>>> he wants the archetype that builds the ‘metron parser plugin’.
>>>
>>> Which is why I hesitate to put out an archetype that is going to
>>> obsolete so soon.
>>>
>>> Does that make sense?
>>>
>>> On March 10, 2017 at 14:50:55, Casey Stella (ceste...@gmail.com) wrote:
>>>
>>> I'm a little concerned about this increasing the size and length of the
>>> build due to the repeated shading. Should we figure out a way to deploy
>>> jars with provided dependencies on metron-parser-common as suggested in
>>> the
>>> previous JIRAs first?
>>>
>>> On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley 
>>> wrote:
>>>
>>> > It sounds like:
>>> > - This is a self-contained chunk of work, that can be tested, reviewed,
>>> > and committed on its own, then the other ideas you propose can follow
>>> it.
>>> > - It crosses a lot of lines, and restructures a lot of code, so will
>>> “rot”
>>> > fairly quickly as other people make commits, so if possible you should
>>> get
>>> > a PR out there and we should work through it as soon as possible.
>>> > Are those both true?
>>> >
>>> > How do other people feel about grouping a given sensor’s parser,
>>> enricher,
>>> > indexing logic all together? It seems to have multiple advantages are
>>> > there also disadvantages?
>>> >
>>> > On 3/10/17, 6:31 AM, "Otto Fowler"  wrote:
>>> >
>>> > As previously discussed here, I have been working on side loading of
>>> > parsers. The goals of this work are:
>>> > * Make it possible of developers to create, maintain and deploy parsers
>>> > outside of the Metron code tree and not have to fork
>>> > * Create maven archetype support for developers of parsers
>>> > * Introduce a parser ‘lifecycle’ to support multiple instances and
>>> > configurations, states of being installed, under configuration, and
>>> > deployed
>>> > etc.
>>> >
>>> > I would like to have some discussion based on where I am after rebasing
>>> > onto METRON-671 which revamps deployment to be totally ambari based.
>>> >
>>> >
>>> > Parsers as components:
>>> >
>>> > I have all the parsers broken out into individual packages/rpms/jars.
>>> > What I have done is taken metron-parsers and broken it out to:
>>> >
>>> > * metron-parsers-common
>>> > * This has all the base classes and interfaces, common testing
>>> > components
>>> > etc
>>> > * metron-parser-base
>>> > * This has the Grok, CSV, and JsonMap parsers and support
>>> > * metron-parser-X
>>> > * A module per parser type which we currently have in the system
>>> > * Each parser has all the indexing, enrichment and parser
>>> > configurations
>>> > for that parser in its package
>>> >
>>> > I will go into packaging and deployment issues in another email.
>>> >
>>> > I have this all working:
>>> > * the parsers are built
>>> > * the parsers are tested
>>> > * the parsers are integrated into the deployment build such that
>>> > vagrant up
>>> > just 

Re: [Discuss] SIDELOADING PARSERS: Parsers as components

[Otto Fowler]

Also a Nar can depend on ‘one’ other nar, which is interesting


On March 10, 2017 at 16:02:18, Otto Fowler (ottobackwa...@gmail.com) wrote:

The isolation is just a ‘extra’ in the parser case.

The parts of Nar that *are* more pertinent:

* supporting a deployment artifact with just a jar, or a tar.gz with a jar
and jar dependencies in it
* taking a ‘package’ and deploying it for loading ( which will upgrade the
deployed part if it is newer )
* setting up the classloader hierarchy between the ‘system’ and provided
things, and the dependencies of the individual plugin



On March 10, 2017 at 15:56:08, Casey Stella (ceste...@gmail.com) wrote:

Why would we need classpath isolation here in the case of the parser?

On Fri, Mar 10, 2017 at 3:55 PM, Otto Fowler 
wrote:

> I *would* use the classloader part, extending it with VFS.
>
>
>
> On March 10, 2017 at 15:53:05, Casey Stella (ceste...@gmail.com) wrote:
>
> I'm a bit worried about copying and pasting from the NiFi project their
> nar infrastructure.  That seems..unclean to me and since we're not using
> the classloader part of nar for this, does it make more sense to just use
> jar?
>
> On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler 
> wrote:
>
>> Compared to how much time vagrant up takes now, you won’t even notice it
>> ;)
>>
>> That is definitely an option.  I guess what I want to work out is if we
>> are going to want to
>> go to NAR, why not just go to NAR.
>>
>> In the end, the customer for this - like Jon Zeolla, isn’t going to care
>> about the intermediate step,
>> he wants the archetype that builds the ‘metron parser plugin’.
>>
>> Which is why I hesitate to put out an archetype that is going to obsolete
>> so soon.
>>
>> Does that make sense?
>>
>> On March 10, 2017 at 14:50:55, Casey Stella (ceste...@gmail.com) wrote:
>>
>> I'm a little concerned about this increasing the size and length of the
>> build due to the repeated shading. Should we figure out a way to deploy
>> jars with provided dependencies on metron-parser-common as suggested in
>> the
>> previous JIRAs first?
>>
>> On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley 
>> wrote:
>>
>> > It sounds like:
>> > - This is a self-contained chunk of work, that can be tested, reviewed,
>> > and committed on its own, then the other ideas you propose can follow
>> it.
>> > - It crosses a lot of lines, and restructures a lot of code, so will
>> “rot”
>> > fairly quickly as other people make commits, so if possible you should
>> get
>> > a PR out there and we should work through it as soon as possible.
>> > Are those both true?
>> >
>> > How do other people feel about grouping a given sensor’s parser,
>> enricher,
>> > indexing logic all together? It seems to have multiple advantages are
>> > there also disadvantages?
>> >
>> > On 3/10/17, 6:31 AM, "Otto Fowler"  wrote:
>> >
>> > As previously discussed here, I have been working on side loading of
>> > parsers. The goals of this work are:
>> > * Make it possible of developers to create, maintain and deploy parsers
>> > outside of the Metron code tree and not have to fork
>> > * Create maven archetype support for developers of parsers
>> > * Introduce a parser ‘lifecycle’ to support multiple instances and
>> > configurations, states of being installed, under configuration, and
>> > deployed
>> > etc.
>> >
>> > I would like to have some discussion based on where I am after rebasing
>> > onto METRON-671 which revamps deployment to be totally ambari based.
>> >
>> >
>> > Parsers as components:
>> >
>> > I have all the parsers broken out into individual packages/rpms/jars.
>> > What I have done is taken metron-parsers and broken it out to:
>> >
>> > * metron-parsers-common
>> > * This has all the base classes and interfaces, common testing
>> > components
>> > etc
>> > * metron-parser-base
>> > * This has the Grok, CSV, and JsonMap parsers and support
>> > * metron-parser-X
>> > * A module per parser type which we currently have in the system
>> > * Each parser has all the indexing, enrichment and parser
>> > configurations
>> > for that parser in its package
>> >
>> > I will go into packaging and deployment issues in another email.
>> >
>> > I have this all working:
>> > * the parsers are built
>> > * the parsers are tested
>> > * the parsers are integrated into the deployment build such that
>> > vagrant up
>> > just works as previously in full and quick dev
>> > * maven component of rpm docker
>> > * the metron.spec file
>> > * ambari installation
>> > * zookeeper configuration deployment
>> > * the ambari parser service code
>> > * the Rest interface works
>> > * see all installed parser configurations etc
>> >
>> >
>> > So this part of the work, is I think ready for a PR and review/next
>> > steps
>> > on it’s own.
>> >
>> > I think that it sets up the components and is a base for building out
>> > the
>> > rest of the functionality we want.
>> >
>> >
>> >

Re: [Discuss] SIDELOADING PARSERS: Parsers as components

[Otto Fowler]

I *would* use the classloader part, extending it with VFS.



On March 10, 2017 at 15:53:05, Casey Stella (ceste...@gmail.com) wrote:

I'm a bit worried about copying and pasting from the NiFi project their nar
infrastructure.  That seems..unclean to me and since we're not using the
classloader part of nar for this, does it make more sense to just use jar?

On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler 
wrote:

> Compared to how much time vagrant up takes now, you won’t even notice it ;)
>
> That is definitely an option.  I guess what I want to work out is if we
> are going to want to
> go to NAR, why not just go to NAR.
>
> In the end, the customer for this - like Jon Zeolla, isn’t going to care
> about the intermediate step,
> he wants the archetype that builds the ‘metron parser plugin’.
>
> Which is why I hesitate to put out an archetype that is going to obsolete
> so soon.
>
> Does that make sense?
>
> On March 10, 2017 at 14:50:55, Casey Stella (ceste...@gmail.com) wrote:
>
> I'm a little concerned about this increasing the size and length of the
> build due to the repeated shading. Should we figure out a way to deploy
> jars with provided dependencies on metron-parser-common as suggested in the
> previous JIRAs first?
>
> On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley 
> wrote:
>
> > It sounds like:
> > - This is a self-contained chunk of work, that can be tested, reviewed,
> > and committed on its own, then the other ideas you propose can follow it.
> > - It crosses a lot of lines, and restructures a lot of code, so will
> “rot”
> > fairly quickly as other people make commits, so if possible you should
> get
> > a PR out there and we should work through it as soon as possible.
> > Are those both true?
> >
> > How do other people feel about grouping a given sensor’s parser,
> enricher,
> > indexing logic all together? It seems to have multiple advantages are
> > there also disadvantages?
> >
> > On 3/10/17, 6:31 AM, "Otto Fowler"  wrote:
> >
> > As previously discussed here, I have been working on side loading of
> > parsers. The goals of this work are:
> > * Make it possible of developers to create, maintain and deploy parsers
> > outside of the Metron code tree and not have to fork
> > * Create maven archetype support for developers of parsers
> > * Introduce a parser ‘lifecycle’ to support multiple instances and
> > configurations, states of being installed, under configuration, and
> > deployed
> > etc.
> >
> > I would like to have some discussion based on where I am after rebasing
> > onto METRON-671 which revamps deployment to be totally ambari based.
> >
> >
> > Parsers as components:
> >
> > I have all the parsers broken out into individual packages/rpms/jars.
> > What I have done is taken metron-parsers and broken it out to:
> >
> > * metron-parsers-common
> > * This has all the base classes and interfaces, common testing
> > components
> > etc
> > * metron-parser-base
> > * This has the Grok, CSV, and JsonMap parsers and support
> > * metron-parser-X
> > * A module per parser type which we currently have in the system
> > * Each parser has all the indexing, enrichment and parser
> > configurations
> > for that parser in its package
> >
> > I will go into packaging and deployment issues in another email.
> >
> > I have this all working:
> > * the parsers are built
> > * the parsers are tested
> > * the parsers are integrated into the deployment build such that
> > vagrant up
> > just works as previously in full and quick dev
> > * maven component of rpm docker
> > * the metron.spec file
> > * ambari installation
> > * zookeeper configuration deployment
> > * the ambari parser service code
> > * the Rest interface works
> > * see all installed parser configurations etc
> >
> >
> > So this part of the work, is I think ready for a PR and review/next
> > steps
> > on it’s own.
> >
> > I think that it sets up the components and is a base for building out
> > the
> > rest of the functionality we want.
> >
> >
> >
>
>

Re: [Discuss] SIDELOADING PARSERS: Parsers as components

[Otto Fowler]

The isolation is just a ‘extra’ in the parser case.

The parts of Nar that *are* more pertinent:

* supporting a deployment artifact with just a jar, or a tar.gz with a jar
and jar dependencies in it
* taking a ‘package’ and deploying it for loading ( which will upgrade the
deployed part if it is newer )
* setting up the classloader hierarchy between the ‘system’ and provided
things, and the dependencies of the individual plugin



On March 10, 2017 at 15:56:08, Casey Stella (ceste...@gmail.com) wrote:

Why would we need classpath isolation here in the case of the parser?

On Fri, Mar 10, 2017 at 3:55 PM, Otto Fowler 
wrote:

> I *would* use the classloader part, extending it with VFS.
>
>
>
> On March 10, 2017 at 15:53:05, Casey Stella (ceste...@gmail.com) wrote:
>
> I'm a bit worried about copying and pasting from the NiFi project their
> nar infrastructure.  That seems..unclean to me and since we're not using
> the classloader part of nar for this, does it make more sense to just use
> jar?
>
> On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler 
> wrote:
>
>> Compared to how much time vagrant up takes now, you won’t even notice it
>> ;)
>>
>> That is definitely an option.  I guess what I want to work out is if we
>> are going to want to
>> go to NAR, why not just go to NAR.
>>
>> In the end, the customer for this - like Jon Zeolla, isn’t going to care
>> about the intermediate step,
>> he wants the archetype that builds the ‘metron parser plugin’.
>>
>> Which is why I hesitate to put out an archetype that is going to obsolete
>> so soon.
>>
>> Does that make sense?
>>
>> On March 10, 2017 at 14:50:55, Casey Stella (ceste...@gmail.com) wrote:
>>
>> I'm a little concerned about this increasing the size and length of the
>> build due to the repeated shading. Should we figure out a way to deploy
>> jars with provided dependencies on metron-parser-common as suggested in
>> the
>> previous JIRAs first?
>>
>> On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley 
>> wrote:
>>
>> > It sounds like:
>> > - This is a self-contained chunk of work, that can be tested, reviewed,
>> > and committed on its own, then the other ideas you propose can follow
>> it.
>> > - It crosses a lot of lines, and restructures a lot of code, so will
>> “rot”
>> > fairly quickly as other people make commits, so if possible you should
>> get
>> > a PR out there and we should work through it as soon as possible.
>> > Are those both true?
>> >
>> > How do other people feel about grouping a given sensor’s parser,
>> enricher,
>> > indexing logic all together? It seems to have multiple advantages are
>> > there also disadvantages?
>> >
>> > On 3/10/17, 6:31 AM, "Otto Fowler"  wrote:
>> >
>> > As previously discussed here, I have been working on side loading of
>> > parsers. The goals of this work are:
>> > * Make it possible of developers to create, maintain and deploy parsers
>> > outside of the Metron code tree and not have to fork
>> > * Create maven archetype support for developers of parsers
>> > * Introduce a parser ‘lifecycle’ to support multiple instances and
>> > configurations, states of being installed, under configuration, and
>> > deployed
>> > etc.
>> >
>> > I would like to have some discussion based on where I am after rebasing
>> > onto METRON-671 which revamps deployment to be totally ambari based.
>> >
>> >
>> > Parsers as components:
>> >
>> > I have all the parsers broken out into individual packages/rpms/jars.
>> > What I have done is taken metron-parsers and broken it out to:
>> >
>> > * metron-parsers-common
>> > * This has all the base classes and interfaces, common testing
>> > components
>> > etc
>> > * metron-parser-base
>> > * This has the Grok, CSV, and JsonMap parsers and support
>> > * metron-parser-X
>> > * A module per parser type which we currently have in the system
>> > * Each parser has all the indexing, enrichment and parser
>> > configurations
>> > for that parser in its package
>> >
>> > I will go into packaging and deployment issues in another email.
>> >
>> > I have this all working:
>> > * the parsers are built
>> > * the parsers are tested
>> > * the parsers are integrated into the deployment build such that
>> > vagrant up
>> > just works as previously in full and quick dev
>> > * maven component of rpm docker
>> > * the metron.spec file
>> > * ambari installation
>> > * zookeeper configuration deployment
>> > * the ambari parser service code
>> > * the Rest interface works
>> > * see all installed parser configurations etc
>> >
>> >
>> > So this part of the work, is I think ready for a PR and review/next
>> > steps
>> > on it’s own.
>> >
>> > I think that it sets up the components and is a base for building out
>> > the
>> > rest of the functionality we want.
>> >
>> >
>> >
>>
>>
>

Re: [Discuss] SIDELOADING PARSERS: Parsers as components

[Casey Stella]

I'm a bit worried about copying and pasting from the NiFi project their nar
infrastructure.  That seems..unclean to me and since we're not using the
classloader part of nar for this, does it make more sense to just use jar?

On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler 
wrote:

> Compared to how much time vagrant up takes now, you won’t even notice it ;)
>
> That is definitely an option.  I guess what I want to work out is if we
> are going to want to
> go to NAR, why not just go to NAR.
>
> In the end, the customer for this - like Jon Zeolla, isn’t going to care
> about the intermediate step,
> he wants the archetype that builds the ‘metron parser plugin’.
>
> Which is why I hesitate to put out an archetype that is going to obsolete
> so soon.
>
> Does that make sense?
>
> On March 10, 2017 at 14:50:55, Casey Stella (ceste...@gmail.com) wrote:
>
> I'm a little concerned about this increasing the size and length of the
> build due to the repeated shading. Should we figure out a way to deploy
> jars with provided dependencies on metron-parser-common as suggested in
> the
> previous JIRAs first?
>
> On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley 
> wrote:
>
> > It sounds like:
> > - This is a self-contained chunk of work, that can be tested, reviewed,
> > and committed on its own, then the other ideas you propose can follow
> it.
> > - It crosses a lot of lines, and restructures a lot of code, so will
> “rot”
> > fairly quickly as other people make commits, so if possible you should
> get
> > a PR out there and we should work through it as soon as possible.
> > Are those both true?
> >
> > How do other people feel about grouping a given sensor’s parser,
> enricher,
> > indexing logic all together? It seems to have multiple advantages are
> > there also disadvantages?
> >
> > On 3/10/17, 6:31 AM, "Otto Fowler"  wrote:
> >
> > As previously discussed here, I have been working on side loading of
> > parsers. The goals of this work are:
> > * Make it possible of developers to create, maintain and deploy parsers
> > outside of the Metron code tree and not have to fork
> > * Create maven archetype support for developers of parsers
> > * Introduce a parser ‘lifecycle’ to support multiple instances and
> > configurations, states of being installed, under configuration, and
> > deployed
> > etc.
> >
> > I would like to have some discussion based on where I am after rebasing
> > onto METRON-671 which revamps deployment to be totally ambari based.
> >
> >
> > Parsers as components:
> >
> > I have all the parsers broken out into individual packages/rpms/jars.
> > What I have done is taken metron-parsers and broken it out to:
> >
> > * metron-parsers-common
> > * This has all the base classes and interfaces, common testing
> > components
> > etc
> > * metron-parser-base
> > * This has the Grok, CSV, and JsonMap parsers and support
> > * metron-parser-X
> > * A module per parser type which we currently have in the system
> > * Each parser has all the indexing, enrichment and parser
> > configurations
> > for that parser in its package
> >
> > I will go into packaging and deployment issues in another email.
> >
> > I have this all working:
> > * the parsers are built
> > * the parsers are tested
> > * the parsers are integrated into the deployment build such that
> > vagrant up
> > just works as previously in full and quick dev
> > * maven component of rpm docker
> > * the metron.spec file
> > * ambari installation
> > * zookeeper configuration deployment
> > * the ambari parser service code
> > * the Rest interface works
> > * see all installed parser configurations etc
> >
> >
> > So this part of the work, is I think ready for a PR and review/next
> > steps
> > on it’s own.
> >
> > I think that it sets up the components and is a base for building out
> > the
> > rest of the functionality we want.
> >
> >
> >
>
>

Re: [Discuss] SIDELOADING PARSERS: Parsers as components

[Casey Stella]

Why would we need classpath isolation here in the case of the parser?

On Fri, Mar 10, 2017 at 3:55 PM, Otto Fowler 
wrote:

> I *would* use the classloader part, extending it with VFS.
>
>
>
> On March 10, 2017 at 15:53:05, Casey Stella (ceste...@gmail.com) wrote:
>
> I'm a bit worried about copying and pasting from the NiFi project their
> nar infrastructure.  That seems..unclean to me and since we're not using
> the classloader part of nar for this, does it make more sense to just use
> jar?
>
> On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler 
> wrote:
>
>> Compared to how much time vagrant up takes now, you won’t even notice it
>> ;)
>>
>> That is definitely an option.  I guess what I want to work out is if we
>> are going to want to
>> go to NAR, why not just go to NAR.
>>
>> In the end, the customer for this - like Jon Zeolla, isn’t going to care
>> about the intermediate step,
>> he wants the archetype that builds the ‘metron parser plugin’.
>>
>> Which is why I hesitate to put out an archetype that is going to obsolete
>> so soon.
>>
>> Does that make sense?
>>
>> On March 10, 2017 at 14:50:55, Casey Stella (ceste...@gmail.com) wrote:
>>
>> I'm a little concerned about this increasing the size and length of the
>> build due to the repeated shading. Should we figure out a way to deploy
>> jars with provided dependencies on metron-parser-common as suggested in
>> the
>> previous JIRAs first?
>>
>> On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley 
>> wrote:
>>
>> > It sounds like:
>> > - This is a self-contained chunk of work, that can be tested, reviewed,
>> > and committed on its own, then the other ideas you propose can follow
>> it.
>> > - It crosses a lot of lines, and restructures a lot of code, so will
>> “rot”
>> > fairly quickly as other people make commits, so if possible you should
>> get
>> > a PR out there and we should work through it as soon as possible.
>> > Are those both true?
>> >
>> > How do other people feel about grouping a given sensor’s parser,
>> enricher,
>> > indexing logic all together? It seems to have multiple advantages are
>> > there also disadvantages?
>> >
>> > On 3/10/17, 6:31 AM, "Otto Fowler"  wrote:
>> >
>> > As previously discussed here, I have been working on side loading of
>> > parsers. The goals of this work are:
>> > * Make it possible of developers to create, maintain and deploy parsers
>> > outside of the Metron code tree and not have to fork
>> > * Create maven archetype support for developers of parsers
>> > * Introduce a parser ‘lifecycle’ to support multiple instances and
>> > configurations, states of being installed, under configuration, and
>> > deployed
>> > etc.
>> >
>> > I would like to have some discussion based on where I am after rebasing
>> > onto METRON-671 which revamps deployment to be totally ambari based.
>> >
>> >
>> > Parsers as components:
>> >
>> > I have all the parsers broken out into individual packages/rpms/jars.
>> > What I have done is taken metron-parsers and broken it out to:
>> >
>> > * metron-parsers-common
>> > * This has all the base classes and interfaces, common testing
>> > components
>> > etc
>> > * metron-parser-base
>> > * This has the Grok, CSV, and JsonMap parsers and support
>> > * metron-parser-X
>> > * A module per parser type which we currently have in the system
>> > * Each parser has all the indexing, enrichment and parser
>> > configurations
>> > for that parser in its package
>> >
>> > I will go into packaging and deployment issues in another email.
>> >
>> > I have this all working:
>> > * the parsers are built
>> > * the parsers are tested
>> > * the parsers are integrated into the deployment build such that
>> > vagrant up
>> > just works as previously in full and quick dev
>> > * maven component of rpm docker
>> > * the metron.spec file
>> > * ambari installation
>> > * zookeeper configuration deployment
>> > * the ambari parser service code
>> > * the Rest interface works
>> > * see all installed parser configurations etc
>> >
>> >
>> > So this part of the work, is I think ready for a PR and review/next
>> > steps
>> > on it’s own.
>> >
>> > I think that it sets up the components and is a base for building out
>> > the
>> > rest of the functionality we want.
>> >
>> >
>> >
>>
>>
>

Re: [Discuss] SIDELOADING PARSERS: Parsers as components

[Casey Stella]

Also, it's not clear that we're ever going to need the classloader bits
from nar for parsers since they are naturally isolated by storm topology.
I might be wrong there though; do you see a scenario?

On Fri, Mar 10, 2017 at 3:53 PM, Casey Stella  wrote:

> I'm a bit worried about copying and pasting from the NiFi project their
> nar infrastructure.  That seems..unclean to me and since we're not using
> the classloader part of nar for this, does it make more sense to just use
> jar?
>
> On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler 
> wrote:
>
>> Compared to how much time vagrant up takes now, you won’t even notice it
>> ;)
>>
>> That is definitely an option.  I guess what I want to work out is if we
>> are going to want to
>> go to NAR, why not just go to NAR.
>>
>> In the end, the customer for this - like Jon Zeolla, isn’t going to care
>> about the intermediate step,
>> he wants the archetype that builds the ‘metron parser plugin’.
>>
>> Which is why I hesitate to put out an archetype that is going to obsolete
>> so soon.
>>
>> Does that make sense?
>>
>> On March 10, 2017 at 14:50:55, Casey Stella (ceste...@gmail.com) wrote:
>>
>> I'm a little concerned about this increasing the size and length of the
>> build due to the repeated shading. Should we figure out a way to deploy
>> jars with provided dependencies on metron-parser-common as suggested in
>> the
>> previous JIRAs first?
>>
>> On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley 
>> wrote:
>>
>> > It sounds like:
>> > - This is a self-contained chunk of work, that can be tested, reviewed,
>> > and committed on its own, then the other ideas you propose can follow
>> it.
>> > - It crosses a lot of lines, and restructures a lot of code, so will
>> “rot”
>> > fairly quickly as other people make commits, so if possible you should
>> get
>> > a PR out there and we should work through it as soon as possible.
>> > Are those both true?
>> >
>> > How do other people feel about grouping a given sensor’s parser,
>> enricher,
>> > indexing logic all together? It seems to have multiple advantages are
>> > there also disadvantages?
>> >
>> > On 3/10/17, 6:31 AM, "Otto Fowler"  wrote:
>> >
>> > As previously discussed here, I have been working on side loading of
>> > parsers. The goals of this work are:
>> > * Make it possible of developers to create, maintain and deploy parsers
>> > outside of the Metron code tree and not have to fork
>> > * Create maven archetype support for developers of parsers
>> > * Introduce a parser ‘lifecycle’ to support multiple instances and
>> > configurations, states of being installed, under configuration, and
>> > deployed
>> > etc.
>> >
>> > I would like to have some discussion based on where I am after rebasing
>> > onto METRON-671 which revamps deployment to be totally ambari based.
>> >
>> >
>> > Parsers as components:
>> >
>> > I have all the parsers broken out into individual packages/rpms/jars.
>> > What I have done is taken metron-parsers and broken it out to:
>> >
>> > * metron-parsers-common
>> > * This has all the base classes and interfaces, common testing
>> > components
>> > etc
>> > * metron-parser-base
>> > * This has the Grok, CSV, and JsonMap parsers and support
>> > * metron-parser-X
>> > * A module per parser type which we currently have in the system
>> > * Each parser has all the indexing, enrichment and parser
>> > configurations
>> > for that parser in its package
>> >
>> > I will go into packaging and deployment issues in another email.
>> >
>> > I have this all working:
>> > * the parsers are built
>> > * the parsers are tested
>> > * the parsers are integrated into the deployment build such that
>> > vagrant up
>> > just works as previously in full and quick dev
>> > * maven component of rpm docker
>> > * the metron.spec file
>> > * ambari installation
>> > * zookeeper configuration deployment
>> > * the ambari parser service code
>> > * the Rest interface works
>> > * see all installed parser configurations etc
>> >
>> >
>> > So this part of the work, is I think ready for a PR and review/next
>> > steps
>> > on it’s own.
>> >
>> > I think that it sets up the components and is a base for building out
>> > the
>> > rest of the functionality we want.
>> >
>> >
>> >
>>
>>
>

Re: [Discuss] SIDELOADING PARSERS: Parsers as components

[Otto Fowler]

Compared to how much time vagrant up takes now, you won’t even notice it ;)

That is definitely an option.  I guess what I want to work out is if we are
going to want to
go to NAR, why not just go to NAR.

In the end, the customer for this - like Jon Zeolla, isn’t going to care
about the intermediate step,
he wants the archetype that builds the ‘metron parser plugin’.

Which is why I hesitate to put out an archetype that is going to obsolete
so soon.

Does that make sense?

On March 10, 2017 at 14:50:55, Casey Stella (ceste...@gmail.com) wrote:

I'm a little concerned about this increasing the size and length of the
build due to the repeated shading. Should we figure out a way to deploy
jars with provided dependencies on metron-parser-common as suggested in the
previous JIRAs first?

On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley  wrote:

> It sounds like:
> - This is a self-contained chunk of work, that can be tested, reviewed,
> and committed on its own, then the other ideas you propose can follow it.
> - It crosses a lot of lines, and restructures a lot of code, so will
“rot”
> fairly quickly as other people make commits, so if possible you should
get
> a PR out there and we should work through it as soon as possible.
> Are those both true?
>
> How do other people feel about grouping a given sensor’s parser,
enricher,
> indexing logic all together? It seems to have multiple advantages are
> there also disadvantages?
>
> On 3/10/17, 6:31 AM, "Otto Fowler"  wrote:
>
> As previously discussed here, I have been working on side loading of
> parsers. The goals of this work are:
> * Make it possible of developers to create, maintain and deploy parsers
> outside of the Metron code tree and not have to fork
> * Create maven archetype support for developers of parsers
> * Introduce a parser ‘lifecycle’ to support multiple instances and
> configurations, states of being installed, under configuration, and
> deployed
> etc.
>
> I would like to have some discussion based on where I am after rebasing
> onto METRON-671 which revamps deployment to be totally ambari based.
>
>
> Parsers as components:
>
> I have all the parsers broken out into individual packages/rpms/jars.
> What I have done is taken metron-parsers and broken it out to:
>
> * metron-parsers-common
> * This has all the base classes and interfaces, common testing
> components
> etc
> * metron-parser-base
> * This has the Grok, CSV, and JsonMap parsers and support
> * metron-parser-X
> * A module per parser type which we currently have in the system
> * Each parser has all the indexing, enrichment and parser
> configurations
> for that parser in its package
>
> I will go into packaging and deployment issues in another email.
>
> I have this all working:
> * the parsers are built
> * the parsers are tested
> * the parsers are integrated into the deployment build such that
> vagrant up
> just works as previously in full and quick dev
> * maven component of rpm docker
> * the metron.spec file
> * ambari installation
> * zookeeper configuration deployment
> * the ambari parser service code
> * the Rest interface works
> * see all installed parser configurations etc
>
>
> So this part of the work, is I think ready for a PR and review/next
> steps
> on it’s own.
>
> I think that it sets up the components and is a base for building out
> the
> rest of the functionality we want.
>
>
>

Re: [Discuss] SIDELOADING PARSERS: Parsers as components

[Casey Stella]

I'm a little concerned about this increasing the size and length of the
build due to the repeated shading.  Should we figure out a way to deploy
jars with provided dependencies on metron-parser-common as suggested in the
previous JIRAs first?

On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley  wrote:

> It sounds like:
> - This is a self-contained chunk of work, that can be tested, reviewed,
> and committed on its own, then the other ideas you propose can follow it.
> - It crosses a lot of lines, and restructures a lot of code, so will “rot”
> fairly quickly as other people make commits, so if possible you should get
> a PR out there and we should work through it as soon as possible.
> Are those both true?
>
> How do other people feel about grouping a given sensor’s parser, enricher,
> indexing logic all together?  It seems to have multiple advantages are
> there also disadvantages?
>
> On 3/10/17, 6:31 AM, "Otto Fowler"  wrote:
>
> As previously discussed here, I have been working on side loading of
> parsers.  The goals of this work are:
> * Make it possible of developers to create, maintain and deploy parsers
> outside of the Metron code tree and not have to fork
> * Create maven archetype support for developers of parsers
> * Introduce a parser ‘lifecycle’ to support multiple instances and
> configurations, states of being installed, under configuration, and
> deployed
> etc.
>
> I would like to have some discussion based on where I am after rebasing
> onto METRON-671 which revamps deployment to be totally ambari based.
>
>
> Parsers as components:
>
> I have all the parsers broken out into individual packages/rpms/jars.
> What I have done is taken metron-parsers and broken it out to:
>
> * metron-parsers-common
> * This has all the base classes and interfaces, common testing
> components
> etc
> * metron-parser-base
> * This has the Grok, CSV, and JsonMap parsers and support
> * metron-parser-X
> * A module per parser type which we currently have in the system
> * Each parser has all the indexing, enrichment and parser
> configurations
> for that parser in its package
>
> I will go into packaging and deployment issues in another email.
>
> I have this all working:
> * the parsers are built
> * the parsers are tested
> * the parsers are integrated into the deployment build such that
> vagrant up
> just works as previously in full and quick dev
> * maven component of rpm docker
>   * the metron.spec file
> * ambari installation
> * zookeeper configuration deployment
> * the ambari parser service code
> * the Rest interface works
> * see all installed parser configurations etc
>
>
> So this part of the work, is I think ready for a PR and review/next
> steps
> on it’s own.
>
> I think that it sets up the components and is a base for building out
> the
> rest of the functionality we want.
>
>
>

Re: [Discuss] SIDELOADING PARSERS: Parsers as components

[Matt Foley]

It sounds like:
- This is a self-contained chunk of work, that can be tested, reviewed, and 
committed on its own, then the other ideas you propose can follow it.
- It crosses a lot of lines, and restructures a lot of code, so will “rot” 
fairly quickly as other people make commits, so if possible you should get a PR 
out there and we should work through it as soon as possible.
Are those both true?

How do other people feel about grouping a given sensor’s parser, enricher, 
indexing logic all together?  It seems to have multiple advantages are there 
also disadvantages?

On 3/10/17, 6:31 AM, "Otto Fowler"  wrote:

As previously discussed here, I have been working on side loading of
parsers.  The goals of this work are:
* Make it possible of developers to create, maintain and deploy parsers
outside of the Metron code tree and not have to fork
* Create maven archetype support for developers of parsers
* Introduce a parser ‘lifecycle’ to support multiple instances and
configurations, states of being installed, under configuration, and deployed
etc.

I would like to have some discussion based on where I am after rebasing
onto METRON-671 which revamps deployment to be totally ambari based.


Parsers as components:

I have all the parsers broken out into individual packages/rpms/jars.
What I have done is taken metron-parsers and broken it out to:

* metron-parsers-common
* This has all the base classes and interfaces, common testing components
etc
* metron-parser-base
* This has the Grok, CSV, and JsonMap parsers and support
* metron-parser-X
* A module per parser type which we currently have in the system
* Each parser has all the indexing, enrichment and parser configurations
for that parser in its package

I will go into packaging and deployment issues in another email.

I have this all working:
* the parsers are built
* the parsers are tested
* the parsers are integrated into the deployment build such that vagrant up
just works as previously in full and quick dev
* maven component of rpm docker
  * the metron.spec file
* ambari installation
* zookeeper configuration deployment
* the ambari parser service code
* the Rest interface works
* see all installed parser configurations etc


So this part of the work, is I think ready for a PR and review/next steps
on it’s own.

I think that it sets up the components and is a base for building out the
rest of the functionality we want.

[Discuss] SIDELOADING PARSERS: Parsers as components

[Otto Fowler]

As previously discussed here, I have been working on side loading of
parsers.  The goals of this work are:
* Make it possible of developers to create, maintain and deploy parsers
outside of the Metron code tree and not have to fork
* Create maven archetype support for developers of parsers
* Introduce a parser ‘lifecycle’ to support multiple instances and
configurations, states of being installed, under configuration, and deployed
etc.

I would like to have some discussion based on where I am after rebasing
onto METRON-671 which revamps deployment to be totally ambari based.


Parsers as components:

I have all the parsers broken out into individual packages/rpms/jars.
What I have done is taken metron-parsers and broken it out to:

* metron-parsers-common
* This has all the base classes and interfaces, common testing components
etc
* metron-parser-base
* This has the Grok, CSV, and JsonMap parsers and support
* metron-parser-X
* A module per parser type which we currently have in the system
* Each parser has all the indexing, enrichment and parser configurations
for that parser in its package

I will go into packaging and deployment issues in another email.

I have this all working:
* the parsers are built
* the parsers are tested
* the parsers are integrated into the deployment build such that vagrant up
just works as previously in full and quick dev
* maven component of rpm docker
  * the metron.spec file
* ambari installation
* zookeeper configuration deployment
* the ambari parser service code
* the Rest interface works
* see all installed parser configurations etc


So this part of the work, is I think ready for a PR and review/next steps
on it’s own.

I think that it sets up the components and is a base for building out the
rest of the functionality we want.