Re: [DISCUSS] Pcap panel architecture

[Michael Miklavcic]

Sweet! That's great news. The pom changes are a lot simpler than I
expected. Very nice.

On Tue, May 8, 2018 at 4:35 PM, Ryan Merriman  wrote:

> Finally figured it out.  Commit is here:
> https://github.com/merrimanr/incubator-metron/commit/
> 22fe5e9ff3c167b42ebeb7a9f1000753a409aff1
>
> It came down to figuring out the right combination of maven dependencies
> and passing in the HDP version to REST as a Java system property.  I also
> included some HDFS setup tasks.  I tested this in full dev and can now
> successfully run a pcap query and get results.  All you should have to do
> is generate some pcap data first.
>
> On Tue, May 8, 2018 at 4:17 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > @Ryan - pulled your branch and experimented with a few things. In doing
> so,
> > it dawned on me that by adding the yarn and hadoop classpath, you
> probably
> > didn't introduce a new classpath issue, rather you probably just moved
> onto
> > the next classpath issue, ie hbase per your exception about hbase jaxb.
> > Anyhow, I put up a branch with some pom changes worth trying in
> conjunction
> > with invoking the rest app startup via "/usr/bin/yarn jar"
> >
> > https://github.com/mmiklavc/metron/tree/ryan-rest-test
> >
> > https://github.com/mmiklavc/metron/commit/5ca23580fc6e043fafae2327c80b65
> > b20ca1c0c9
> >
> > Mike
> >
> >
> > On Tue, May 8, 2018 at 7:44 AM, Simon Elliston Ball <
> > si...@simonellistonball.com> wrote:
> >
> > > That would be a step closer to something more like a micro-service
> > > architecture. However, I would want to make sure we think about the
> > > operational complexity, and mpack implications of having another server
> > > installed and running somewhere on the cluster (also, ssl, kerberos,
> etc
> > > etc requirements for that service).
> > >
> > > On 8 May 2018 at 14:27, Ryan Merriman  wrote:
> > >
> > > > +1 to having metron-api as it's own service and using a gateway type
> > > > pattern.
> > > >
> > > > On Tue, May 8, 2018 at 8:13 AM, Otto Fowler  >
> > > > wrote:
> > > >
> > > > > Why not have metron-api as it’s own service and use a ‘gateway’
> type
> > > > > pattern in rest?
> > > > >
> > > > >
> > > > > On May 8, 2018 at 08:45:33, Ryan Merriman (merrim...@gmail.com)
> > wrote:
> > > > >
> > > > > Moving the yarn classpath command earlier in the classpath now
> gives
> > > this
> > > > > error:
> > > > >
> > > > > Caused by: java.lang.NoSuchMethodError:
> > > > > javax.servlet.ServletContext.getVirtualServerName()Ljava/
> > lang/String;
> > > > >
> > > > > I will experiment with other combinations, I suspect we will need
> > > > > finer-grain control over the order.
> > > > >
> > > > > The grep matches class names inside jar files. I use this all the
> > time
> > > > and
> > > > > it's really useful.
> > > > >
> > > > > The metron-rest jar is already shaded.
> > > > >
> > > > > Reverse engineering the yarn jar command was the next thing I was
> > going
> > > > to
> > > > > try. Will let you know how it goes.
> > > > >
> > > > > On Tue, May 8, 2018 at 12:36 AM, Michael Miklavcic <
> > > > > michael.miklav...@gmail.com> wrote:
> > > > >
> > > > > > What order did you add the hadoop or yarn classpath? The "shaded"
> > > > > package
> > > > > > stands out to me in this name "org.apache.hadoop.hbase.*shaded*
> > > > > > .org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider." Maybe try
> > > adding
> > > > > > those packages earlier on the classpath.
> > > > > >
> > > > > > I think that find command needs a "jar tvf", otherwise you're
> > looking
> > > > > for a
> > > > > > class name in jar file names.
> > > > > >
> > > > > > Have you tried shading the rest jar?
> > > > > >
> > > > > > I'd also look at the classpath you get when running "yarn jar" to
> > > start
> > > > > the
> > > > > > existing pcap service, per the instructions in
> > metron-api/README.md.
> > > > > >
> > > > > >
> > > > > > On Mon, May 7, 2018 at 3:28 PM, Ryan Merriman <
> merrim...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > To explore the idea of merging metron-api into metron-rest and
> > > > running
> > > > > > pcap
> > > > > > > queries inside our REST application, I created a simple test
> > here:
> > > > > > > https://github.com/merrimanr/incubator-metron/tree/pcap-
> > rest-test.
> > > A
> > > > > > > summary of what's included:
> > > > > > >
> > > > > > > - Added pcap as a dependency in the metron-rest pom.xml
> > > > > > > - Added a pcap query controller endpoint at
> > > > > > > http://node1:8082/swagger-ui.html#!/pcap-query-controller/
> > > > > > queryUsingGET
> > > > > > > - Added a pcap query service that runs a simple, hardcoded
> query
> > > > > > >
> > > > > > > Generate some pcap data using pycapa (
> > > > > > > https://github.com/apache/metron/tree/master/metron-
> > sensors/pycapa
> > > )
> > > > > and
> > > > > > > the
> > > > > > > pcap topology (
> > > > > > > 

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

Finally figured it out.  Commit is here:
https://github.com/merrimanr/incubator-metron/commit/22fe5e9ff3c167b42ebeb7a9f1000753a409aff1

It came down to figuring out the right combination of maven dependencies
and passing in the HDP version to REST as a Java system property.  I also
included some HDFS setup tasks.  I tested this in full dev and can now
successfully run a pcap query and get results.  All you should have to do
is generate some pcap data first.

On Tue, May 8, 2018 at 4:17 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> @Ryan - pulled your branch and experimented with a few things. In doing so,
> it dawned on me that by adding the yarn and hadoop classpath, you probably
> didn't introduce a new classpath issue, rather you probably just moved onto
> the next classpath issue, ie hbase per your exception about hbase jaxb.
> Anyhow, I put up a branch with some pom changes worth trying in conjunction
> with invoking the rest app startup via "/usr/bin/yarn jar"
>
> https://github.com/mmiklavc/metron/tree/ryan-rest-test
>
> https://github.com/mmiklavc/metron/commit/5ca23580fc6e043fafae2327c80b65
> b20ca1c0c9
>
> Mike
>
>
> On Tue, May 8, 2018 at 7:44 AM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> > That would be a step closer to something more like a micro-service
> > architecture. However, I would want to make sure we think about the
> > operational complexity, and mpack implications of having another server
> > installed and running somewhere on the cluster (also, ssl, kerberos, etc
> > etc requirements for that service).
> >
> > On 8 May 2018 at 14:27, Ryan Merriman  wrote:
> >
> > > +1 to having metron-api as it's own service and using a gateway type
> > > pattern.
> > >
> > > On Tue, May 8, 2018 at 8:13 AM, Otto Fowler 
> > > wrote:
> > >
> > > > Why not have metron-api as it’s own service and use a ‘gateway’ type
> > > > pattern in rest?
> > > >
> > > >
> > > > On May 8, 2018 at 08:45:33, Ryan Merriman (merrim...@gmail.com)
> wrote:
> > > >
> > > > Moving the yarn classpath command earlier in the classpath now gives
> > this
> > > > error:
> > > >
> > > > Caused by: java.lang.NoSuchMethodError:
> > > > javax.servlet.ServletContext.getVirtualServerName()Ljava/
> lang/String;
> > > >
> > > > I will experiment with other combinations, I suspect we will need
> > > > finer-grain control over the order.
> > > >
> > > > The grep matches class names inside jar files. I use this all the
> time
> > > and
> > > > it's really useful.
> > > >
> > > > The metron-rest jar is already shaded.
> > > >
> > > > Reverse engineering the yarn jar command was the next thing I was
> going
> > > to
> > > > try. Will let you know how it goes.
> > > >
> > > > On Tue, May 8, 2018 at 12:36 AM, Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > What order did you add the hadoop or yarn classpath? The "shaded"
> > > > package
> > > > > stands out to me in this name "org.apache.hadoop.hbase.*shaded*
> > > > > .org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider." Maybe try
> > adding
> > > > > those packages earlier on the classpath.
> > > > >
> > > > > I think that find command needs a "jar tvf", otherwise you're
> looking
> > > > for a
> > > > > class name in jar file names.
> > > > >
> > > > > Have you tried shading the rest jar?
> > > > >
> > > > > I'd also look at the classpath you get when running "yarn jar" to
> > start
> > > > the
> > > > > existing pcap service, per the instructions in
> metron-api/README.md.
> > > > >
> > > > >
> > > > > On Mon, May 7, 2018 at 3:28 PM, Ryan Merriman  >
> > > > wrote:
> > > > >
> > > > > > To explore the idea of merging metron-api into metron-rest and
> > > running
> > > > > pcap
> > > > > > queries inside our REST application, I created a simple test
> here:
> > > > > > https://github.com/merrimanr/incubator-metron/tree/pcap-
> rest-test.
> > A
> > > > > > summary of what's included:
> > > > > >
> > > > > > - Added pcap as a dependency in the metron-rest pom.xml
> > > > > > - Added a pcap query controller endpoint at
> > > > > > http://node1:8082/swagger-ui.html#!/pcap-query-controller/
> > > > > queryUsingGET
> > > > > > - Added a pcap query service that runs a simple, hardcoded query
> > > > > >
> > > > > > Generate some pcap data using pycapa (
> > > > > > https://github.com/apache/metron/tree/master/metron-
> sensors/pycapa
> > )
> > > > and
> > > > > > the
> > > > > > pcap topology (
> > > > > > https://github.com/apache/metron/tree/master/metron-
> > > > > > platform/metron-pcap-backend#starting-the-topology).
> > > > > > After this initial setup there should be data in HDFS at
> > > > > > "/apps/metron/pcap". I believe this should be enough to exercise
> > the
> > > > > > issue. Just hit the endpoint referenced above. I tested this in
> an
> > > > > > already running full dev by building and deploying the
> metron-rest
> > > > jar.
> > > > > I

Re: [DISCUSS] Pcap panel architecture

[Michael Miklavcic]

@Ryan - pulled your branch and experimented with a few things. In doing so,
it dawned on me that by adding the yarn and hadoop classpath, you probably
didn't introduce a new classpath issue, rather you probably just moved onto
the next classpath issue, ie hbase per your exception about hbase jaxb.
Anyhow, I put up a branch with some pom changes worth trying in conjunction
with invoking the rest app startup via "/usr/bin/yarn jar"

https://github.com/mmiklavc/metron/tree/ryan-rest-test

https://github.com/mmiklavc/metron/commit/5ca23580fc6e043fafae2327c80b65b20ca1c0c9

Mike


On Tue, May 8, 2018 at 7:44 AM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> That would be a step closer to something more like a micro-service
> architecture. However, I would want to make sure we think about the
> operational complexity, and mpack implications of having another server
> installed and running somewhere on the cluster (also, ssl, kerberos, etc
> etc requirements for that service).
>
> On 8 May 2018 at 14:27, Ryan Merriman  wrote:
>
> > +1 to having metron-api as it's own service and using a gateway type
> > pattern.
> >
> > On Tue, May 8, 2018 at 8:13 AM, Otto Fowler 
> > wrote:
> >
> > > Why not have metron-api as it’s own service and use a ‘gateway’ type
> > > pattern in rest?
> > >
> > >
> > > On May 8, 2018 at 08:45:33, Ryan Merriman (merrim...@gmail.com) wrote:
> > >
> > > Moving the yarn classpath command earlier in the classpath now gives
> this
> > > error:
> > >
> > > Caused by: java.lang.NoSuchMethodError:
> > > javax.servlet.ServletContext.getVirtualServerName()Ljava/lang/String;
> > >
> > > I will experiment with other combinations, I suspect we will need
> > > finer-grain control over the order.
> > >
> > > The grep matches class names inside jar files. I use this all the time
> > and
> > > it's really useful.
> > >
> > > The metron-rest jar is already shaded.
> > >
> > > Reverse engineering the yarn jar command was the next thing I was going
> > to
> > > try. Will let you know how it goes.
> > >
> > > On Tue, May 8, 2018 at 12:36 AM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > What order did you add the hadoop or yarn classpath? The "shaded"
> > > package
> > > > stands out to me in this name "org.apache.hadoop.hbase.*shaded*
> > > > .org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider." Maybe try
> adding
> > > > those packages earlier on the classpath.
> > > >
> > > > I think that find command needs a "jar tvf", otherwise you're looking
> > > for a
> > > > class name in jar file names.
> > > >
> > > > Have you tried shading the rest jar?
> > > >
> > > > I'd also look at the classpath you get when running "yarn jar" to
> start
> > > the
> > > > existing pcap service, per the instructions in metron-api/README.md.
> > > >
> > > >
> > > > On Mon, May 7, 2018 at 3:28 PM, Ryan Merriman 
> > > wrote:
> > > >
> > > > > To explore the idea of merging metron-api into metron-rest and
> > running
> > > > pcap
> > > > > queries inside our REST application, I created a simple test here:
> > > > > https://github.com/merrimanr/incubator-metron/tree/pcap-rest-test.
> A
> > > > > summary of what's included:
> > > > >
> > > > > - Added pcap as a dependency in the metron-rest pom.xml
> > > > > - Added a pcap query controller endpoint at
> > > > > http://node1:8082/swagger-ui.html#!/pcap-query-controller/
> > > > queryUsingGET
> > > > > - Added a pcap query service that runs a simple, hardcoded query
> > > > >
> > > > > Generate some pcap data using pycapa (
> > > > > https://github.com/apache/metron/tree/master/metron-sensors/pycapa
> )
> > > and
> > > > > the
> > > > > pcap topology (
> > > > > https://github.com/apache/metron/tree/master/metron-
> > > > > platform/metron-pcap-backend#starting-the-topology).
> > > > > After this initial setup there should be data in HDFS at
> > > > > "/apps/metron/pcap". I believe this should be enough to exercise
> the
> > > > > issue. Just hit the endpoint referenced above. I tested this in an
> > > > > already running full dev by building and deploying the metron-rest
> > > jar.
> > > > I
> > > > > did not rebuild full dev with this change but I would still expect
> it
> > > to
> > > > > work. Let me know if it doesn't.
> > > > >
> > > > > The first error I see when I hit this endpoint is:
> > > > >
> > > > > java.lang.NoClassDefFoundError:
> > > > > org/apache/hadoop/yarn/webapp/YarnJacksonJaxbJsonProvider.
> > > > >
> > > > > Here are the things I've tried so far:
> > > > >
> > > > > - Run the REST application with the YARN jar command since this is
> > how
> > > > > all our other YARN/MR-related applications are started (metron-api,
> > > > > MAAS,
> > > > > pcap query, etc). I wouldn't expect this to work since we have
> > > > runtime
> > > > > dependencies on our shaded elasticsearch and parser jars and I'm
> not
> > > > > aware
> > > > > of a way to add additional 

Re: [DISCUSS] Pcap panel architecture

[Simon Elliston Ball]

That would be a step closer to something more like a micro-service
architecture. However, I would want to make sure we think about the
operational complexity, and mpack implications of having another server
installed and running somewhere on the cluster (also, ssl, kerberos, etc
etc requirements for that service).

On 8 May 2018 at 14:27, Ryan Merriman  wrote:

> +1 to having metron-api as it's own service and using a gateway type
> pattern.
>
> On Tue, May 8, 2018 at 8:13 AM, Otto Fowler 
> wrote:
>
> > Why not have metron-api as it’s own service and use a ‘gateway’ type
> > pattern in rest?
> >
> >
> > On May 8, 2018 at 08:45:33, Ryan Merriman (merrim...@gmail.com) wrote:
> >
> > Moving the yarn classpath command earlier in the classpath now gives this
> > error:
> >
> > Caused by: java.lang.NoSuchMethodError:
> > javax.servlet.ServletContext.getVirtualServerName()Ljava/lang/String;
> >
> > I will experiment with other combinations, I suspect we will need
> > finer-grain control over the order.
> >
> > The grep matches class names inside jar files. I use this all the time
> and
> > it's really useful.
> >
> > The metron-rest jar is already shaded.
> >
> > Reverse engineering the yarn jar command was the next thing I was going
> to
> > try. Will let you know how it goes.
> >
> > On Tue, May 8, 2018 at 12:36 AM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > What order did you add the hadoop or yarn classpath? The "shaded"
> > package
> > > stands out to me in this name "org.apache.hadoop.hbase.*shaded*
> > > .org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider." Maybe try adding
> > > those packages earlier on the classpath.
> > >
> > > I think that find command needs a "jar tvf", otherwise you're looking
> > for a
> > > class name in jar file names.
> > >
> > > Have you tried shading the rest jar?
> > >
> > > I'd also look at the classpath you get when running "yarn jar" to start
> > the
> > > existing pcap service, per the instructions in metron-api/README.md.
> > >
> > >
> > > On Mon, May 7, 2018 at 3:28 PM, Ryan Merriman 
> > wrote:
> > >
> > > > To explore the idea of merging metron-api into metron-rest and
> running
> > > pcap
> > > > queries inside our REST application, I created a simple test here:
> > > > https://github.com/merrimanr/incubator-metron/tree/pcap-rest-test. A
> > > > summary of what's included:
> > > >
> > > > - Added pcap as a dependency in the metron-rest pom.xml
> > > > - Added a pcap query controller endpoint at
> > > > http://node1:8082/swagger-ui.html#!/pcap-query-controller/
> > > queryUsingGET
> > > > - Added a pcap query service that runs a simple, hardcoded query
> > > >
> > > > Generate some pcap data using pycapa (
> > > > https://github.com/apache/metron/tree/master/metron-sensors/pycapa)
> > and
> > > > the
> > > > pcap topology (
> > > > https://github.com/apache/metron/tree/master/metron-
> > > > platform/metron-pcap-backend#starting-the-topology).
> > > > After this initial setup there should be data in HDFS at
> > > > "/apps/metron/pcap". I believe this should be enough to exercise the
> > > > issue. Just hit the endpoint referenced above. I tested this in an
> > > > already running full dev by building and deploying the metron-rest
> > jar.
> > > I
> > > > did not rebuild full dev with this change but I would still expect it
> > to
> > > > work. Let me know if it doesn't.
> > > >
> > > > The first error I see when I hit this endpoint is:
> > > >
> > > > java.lang.NoClassDefFoundError:
> > > > org/apache/hadoop/yarn/webapp/YarnJacksonJaxbJsonProvider.
> > > >
> > > > Here are the things I've tried so far:
> > > >
> > > > - Run the REST application with the YARN jar command since this is
> how
> > > > all our other YARN/MR-related applications are started (metron-api,
> > > > MAAS,
> > > > pcap query, etc). I wouldn't expect this to work since we have
> > > runtime
> > > > dependencies on our shaded elasticsearch and parser jars and I'm not
> > > > aware
> > > > of a way to add additional jars to the classpath with the YARN jar
> > > > command
> > > > (is there a way?). Either way I get this error:
> > > >
> > > > 18/05/04 19:49:56 WARN reflections.Reflections: could not create Dir
> > > using
> > > > jarFile from url file:/usr/hdp/2.6.4.0-91/hadoop/lib/ojdbc6.jar.
> > > skipping.
> > > > java.lang.NullPointerException
> > > >
> > > >
> > > > - I tried adding `yarn classpath` and `hadoop classpath` to the
> > > > classpath in /usr/metron/0.4.3/bin/metron-rest.sh (REST start
> > > > script). I
> > > > get this error:
> > > >
> > > > java.lang.ClassNotFoundException:
> > > > org.apache.hadoop.hbase.shaded.org.codehaus.jackson.
> > > > jaxrs.JacksonJaxbJsonProvider
> > > >
> > > >
> > > > - I searched for the class in the previous attempt but could not find
> > > it
> > > > in full dev:
> > > >
> > > > find / -name "*.jar" 2>/dev/null | xargs grep
> > > > 

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

+1 to having metron-api as it's own service and using a gateway type
pattern.

On Tue, May 8, 2018 at 8:13 AM, Otto Fowler  wrote:

> Why not have metron-api as it’s own service and use a ‘gateway’ type
> pattern in rest?
>
>
> On May 8, 2018 at 08:45:33, Ryan Merriman (merrim...@gmail.com) wrote:
>
> Moving the yarn classpath command earlier in the classpath now gives this
> error:
>
> Caused by: java.lang.NoSuchMethodError:
> javax.servlet.ServletContext.getVirtualServerName()Ljava/lang/String;
>
> I will experiment with other combinations, I suspect we will need
> finer-grain control over the order.
>
> The grep matches class names inside jar files. I use this all the time and
> it's really useful.
>
> The metron-rest jar is already shaded.
>
> Reverse engineering the yarn jar command was the next thing I was going to
> try. Will let you know how it goes.
>
> On Tue, May 8, 2018 at 12:36 AM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > What order did you add the hadoop or yarn classpath? The "shaded"
> package
> > stands out to me in this name "org.apache.hadoop.hbase.*shaded*
> > .org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider." Maybe try adding
> > those packages earlier on the classpath.
> >
> > I think that find command needs a "jar tvf", otherwise you're looking
> for a
> > class name in jar file names.
> >
> > Have you tried shading the rest jar?
> >
> > I'd also look at the classpath you get when running "yarn jar" to start
> the
> > existing pcap service, per the instructions in metron-api/README.md.
> >
> >
> > On Mon, May 7, 2018 at 3:28 PM, Ryan Merriman 
> wrote:
> >
> > > To explore the idea of merging metron-api into metron-rest and running
> > pcap
> > > queries inside our REST application, I created a simple test here:
> > > https://github.com/merrimanr/incubator-metron/tree/pcap-rest-test. A
> > > summary of what's included:
> > >
> > > - Added pcap as a dependency in the metron-rest pom.xml
> > > - Added a pcap query controller endpoint at
> > > http://node1:8082/swagger-ui.html#!/pcap-query-controller/
> > queryUsingGET
> > > - Added a pcap query service that runs a simple, hardcoded query
> > >
> > > Generate some pcap data using pycapa (
> > > https://github.com/apache/metron/tree/master/metron-sensors/pycapa)
> and
> > > the
> > > pcap topology (
> > > https://github.com/apache/metron/tree/master/metron-
> > > platform/metron-pcap-backend#starting-the-topology).
> > > After this initial setup there should be data in HDFS at
> > > "/apps/metron/pcap". I believe this should be enough to exercise the
> > > issue. Just hit the endpoint referenced above. I tested this in an
> > > already running full dev by building and deploying the metron-rest
> jar.
> > I
> > > did not rebuild full dev with this change but I would still expect it
> to
> > > work. Let me know if it doesn't.
> > >
> > > The first error I see when I hit this endpoint is:
> > >
> > > java.lang.NoClassDefFoundError:
> > > org/apache/hadoop/yarn/webapp/YarnJacksonJaxbJsonProvider.
> > >
> > > Here are the things I've tried so far:
> > >
> > > - Run the REST application with the YARN jar command since this is how
> > > all our other YARN/MR-related applications are started (metron-api,
> > > MAAS,
> > > pcap query, etc). I wouldn't expect this to work since we have
> > runtime
> > > dependencies on our shaded elasticsearch and parser jars and I'm not
> > > aware
> > > of a way to add additional jars to the classpath with the YARN jar
> > > command
> > > (is there a way?). Either way I get this error:
> > >
> > > 18/05/04 19:49:56 WARN reflections.Reflections: could not create Dir
> > using
> > > jarFile from url file:/usr/hdp/2.6.4.0-91/hadoop/lib/ojdbc6.jar.
> > skipping.
> > > java.lang.NullPointerException
> > >
> > >
> > > - I tried adding `yarn classpath` and `hadoop classpath` to the
> > > classpath in /usr/metron/0.4.3/bin/metron-rest.sh (REST start
> > > script). I
> > > get this error:
> > >
> > > java.lang.ClassNotFoundException:
> > > org.apache.hadoop.hbase.shaded.org.codehaus.jackson.
> > > jaxrs.JacksonJaxbJsonProvider
> > >
> > >
> > > - I searched for the class in the previous attempt but could not find
> > it
> > > in full dev:
> > >
> > > find / -name "*.jar" 2>/dev/null | xargs grep
> > > org/apache/hadoop/hbase/shaded/org/codehaus/jackson/
> > > jaxrs/JacksonJaxbJsonProvider
> > > 2>/dev/null
> > >
> > >
> > > - Further up in the stack trace I see the error happens when
> > initiating
> > > the org.apache.hadoop.yarn.util.timeline.TimelineUtils class. I
> > tried
> > > setting "yarn.timeline-service.enabled" in Ambari to false and then I
> > > get
> > > this error:
> > >
> > > Unable to parse
> > > '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as
> a
> > > URI, check the setting for mapreduce.application.framework.path
> > >
> > >
> > > - I've tried adding different hadoop, hbase, yarn and mapreduce Maven
> > > 

Re: [DISCUSS] Pcap panel architecture

[Otto Fowler]

Why not have metron-api as it’s own service and use a ‘gateway’ type
pattern in rest?


On May 8, 2018 at 08:45:33, Ryan Merriman (merrim...@gmail.com) wrote:

Moving the yarn classpath command earlier in the classpath now gives this
error:

Caused by: java.lang.NoSuchMethodError:
javax.servlet.ServletContext.getVirtualServerName()Ljava/lang/String;

I will experiment with other combinations, I suspect we will need
finer-grain control over the order.

The grep matches class names inside jar files. I use this all the time and
it's really useful.

The metron-rest jar is already shaded.

Reverse engineering the yarn jar command was the next thing I was going to
try. Will let you know how it goes.

On Tue, May 8, 2018 at 12:36 AM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> What order did you add the hadoop or yarn classpath? The "shaded" package
> stands out to me in this name "org.apache.hadoop.hbase.*shaded*
> .org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider." Maybe try adding
> those packages earlier on the classpath.
>
> I think that find command needs a "jar tvf", otherwise you're looking for
a
> class name in jar file names.
>
> Have you tried shading the rest jar?
>
> I'd also look at the classpath you get when running "yarn jar" to start
the
> existing pcap service, per the instructions in metron-api/README.md.
>
>
> On Mon, May 7, 2018 at 3:28 PM, Ryan Merriman 
wrote:
>
> > To explore the idea of merging metron-api into metron-rest and running
> pcap
> > queries inside our REST application, I created a simple test here:
> > https://github.com/merrimanr/incubator-metron/tree/pcap-rest-test. A
> > summary of what's included:
> >
> > - Added pcap as a dependency in the metron-rest pom.xml
> > - Added a pcap query controller endpoint at
> > http://node1:8082/swagger-ui.html#!/pcap-query-controller/
> queryUsingGET
> > - Added a pcap query service that runs a simple, hardcoded query
> >
> > Generate some pcap data using pycapa (
> > https://github.com/apache/metron/tree/master/metron-sensors/pycapa) and
> > the
> > pcap topology (
> > https://github.com/apache/metron/tree/master/metron-
> > platform/metron-pcap-backend#starting-the-topology).
> > After this initial setup there should be data in HDFS at
> > "/apps/metron/pcap". I believe this should be enough to exercise the
> > issue. Just hit the endpoint referenced above. I tested this in an
> > already running full dev by building and deploying the metron-rest jar.
> I
> > did not rebuild full dev with this change but I would still expect it
to
> > work. Let me know if it doesn't.
> >
> > The first error I see when I hit this endpoint is:
> >
> > java.lang.NoClassDefFoundError:
> > org/apache/hadoop/yarn/webapp/YarnJacksonJaxbJsonProvider.
> >
> > Here are the things I've tried so far:
> >
> > - Run the REST application with the YARN jar command since this is how
> > all our other YARN/MR-related applications are started (metron-api,
> > MAAS,
> > pcap query, etc). I wouldn't expect this to work since we have
> runtime
> > dependencies on our shaded elasticsearch and parser jars and I'm not
> > aware
> > of a way to add additional jars to the classpath with the YARN jar
> > command
> > (is there a way?). Either way I get this error:
> >
> > 18/05/04 19:49:56 WARN reflections.Reflections: could not create Dir
> using
> > jarFile from url file:/usr/hdp/2.6.4.0-91/hadoop/lib/ojdbc6.jar.
> skipping.
> > java.lang.NullPointerException
> >
> >
> > - I tried adding `yarn classpath` and `hadoop classpath` to the
> > classpath in /usr/metron/0.4.3/bin/metron-rest.sh (REST start
> > script). I
> > get this error:
> >
> > java.lang.ClassNotFoundException:
> > org.apache.hadoop.hbase.shaded.org.codehaus.jackson.
> > jaxrs.JacksonJaxbJsonProvider
> >
> >
> > - I searched for the class in the previous attempt but could not find
> it
> > in full dev:
> >
> > find / -name "*.jar" 2>/dev/null | xargs grep
> > org/apache/hadoop/hbase/shaded/org/codehaus/jackson/
> > jaxrs/JacksonJaxbJsonProvider
> > 2>/dev/null
> >
> >
> > - Further up in the stack trace I see the error happens when
> initiating
> > the org.apache.hadoop.yarn.util.timeline.TimelineUtils class. I
> tried
> > setting "yarn.timeline-service.enabled" in Ambari to false and then I
> > get
> > this error:
> >
> > Unable to parse
> > '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a
> > URI, check the setting for mapreduce.application.framework.path
> >
> >
> > - I've tried adding different hadoop, hbase, yarn and mapreduce Maven
> > dependencies without any success
> > - hadoop-yarn-client
> > - hadoop-yarn-common
> > - hadoop-mapreduce-client-core
> > - hadoop-yarn-server-common
> > - hadoop-yarn-api
> > - hbase-server
> >
> > I will keep exploring other possible solutions. Let me know if anyone
> has
> > any ideas.
> >
> > On Mon, May 7, 2018 at 9:02 AM, Otto Fowler 
> > wrote:
> >
> > > I can imagine a new generic 

Re: Streaming Machine Learning use case

[Simon Elliston Ball]

Do you mean Apache SAMOA? I'm not sure of the status of that project, and
it doesn't look particularly lively (last real activity on the lists was 2
months ago, last commits, 7 months ago).

That said, there seem to be some interesting algorithms implemented in
there. The VHT algorithm and the clustering may be relevant, though we have
other efficient means of streaming clustering already in Metron. I would
also argue that we'd be better off looking at algorithms in Spark for
things like frequent pattern mining, though there the FP growth algorithm
is of course primarily a batch implementation.

Are there any SAMOA algorithms in particular that you think would be
relevant to Metron use cases?

Simon


On 8 May 2018 at 07:29, Ali Nazemian  wrote:

> Hi all,
>
> I was wondering if someone has used Metron with any streaming ML framework
> such as SAMOA? I know that Metron provides Machine Learning separately via
> MAAS. However, it is hard to manage it from operational perspective
> especially if we want to have a pretty dynamic and evolving model. SAMOA
> seems to be a very slow project (or maybe even dead). However, it looks
> very close from the integration point of view with Metron, so I wanted to
> see if anyone had tried SAMOA in practice and especially with Metron use
> cases.
>
> Regards,
> Ali
>



-- 
--
simon elliston ball
@sireb

Re: [DISCUSS] Pcap panel architecture

[Ryan Merriman]

Moving the yarn classpath command earlier in the classpath now gives this
error:

Caused by: java.lang.NoSuchMethodError:
javax.servlet.ServletContext.getVirtualServerName()Ljava/lang/String;

I will experiment with other combinations, I suspect we will need
finer-grain control over the order.

The grep matches class names inside jar files.  I use this all the time and
it's really useful.

The metron-rest jar is already shaded.

Reverse engineering the yarn jar command was the next thing I was going to
try.  Will let you know how it goes.

On Tue, May 8, 2018 at 12:36 AM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> What order did you add the hadoop or yarn classpath? The "shaded" package
> stands out to me in this name "org.apache.hadoop.hbase.*shaded*
> .org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider." Maybe try adding
> those packages earlier on the classpath.
>
> I think that find command needs a "jar tvf", otherwise you're looking for a
> class name in jar file names.
>
> Have you tried shading the rest jar?
>
> I'd also look at the classpath you get when running "yarn jar" to start the
> existing pcap service, per the instructions in metron-api/README.md.
>
>
> On Mon, May 7, 2018 at 3:28 PM, Ryan Merriman  wrote:
>
> > To explore the idea of merging metron-api into metron-rest and running
> pcap
> > queries inside our REST application, I created a simple test here:
> > https://github.com/merrimanr/incubator-metron/tree/pcap-rest-test.  A
> > summary of what's included:
> >
> >- Added pcap as a dependency in the metron-rest pom.xml
> >- Added a pcap query controller endpoint at
> >http://node1:8082/swagger-ui.html#!/pcap-query-controller/
> queryUsingGET
> >- Added a pcap query service that runs a simple, hardcoded query
> >
> > Generate some pcap data using pycapa (
> > https://github.com/apache/metron/tree/master/metron-sensors/pycapa) and
> > the
> > pcap topology (
> > https://github.com/apache/metron/tree/master/metron-
> > platform/metron-pcap-backend#starting-the-topology).
> > After this initial setup there should be data in HDFS at
> > "/apps/metron/pcap".  I believe this should be enough to exercise the
> > issue.  Just hit the endpoint referenced above.  I tested this in an
> > already running full dev by building and deploying the metron-rest jar.
> I
> > did not rebuild full dev with this change but I would still expect it to
> > work.  Let me know if it doesn't.
> >
> > The first error I see when I hit this endpoint is:
> >
> > java.lang.NoClassDefFoundError:
> > org/apache/hadoop/yarn/webapp/YarnJacksonJaxbJsonProvider.
> >
> > Here are the things I've tried so far:
> >
> >- Run the REST application with the YARN jar command since this is how
> >all our other YARN/MR-related applications are started (metron-api,
> > MAAS,
> >pcap query, etc).  I wouldn't expect this to work since we have
> runtime
> >dependencies on our shaded elasticsearch and parser jars and I'm not
> > aware
> >of a way to add additional jars to the classpath with the YARN jar
> > command
> >(is there a way?).  Either way I get this error:
> >
> > 18/05/04 19:49:56 WARN reflections.Reflections: could not create Dir
> using
> > jarFile from url file:/usr/hdp/2.6.4.0-91/hadoop/lib/ojdbc6.jar.
> skipping.
> > java.lang.NullPointerException
> >
> >
> >- I tried adding `yarn classpath` and `hadoop classpath` to the
> >classpath in /usr/metron/0.4.3/bin/metron-rest.sh (REST start
> > script).  I
> >get this error:
> >
> > java.lang.ClassNotFoundException:
> > org.apache.hadoop.hbase.shaded.org.codehaus.jackson.
> > jaxrs.JacksonJaxbJsonProvider
> >
> >
> >- I searched for the class in the previous attempt but could not find
> it
> >in full dev:
> >
> > find / -name "*.jar" 2>/dev/null | xargs grep
> > org/apache/hadoop/hbase/shaded/org/codehaus/jackson/
> > jaxrs/JacksonJaxbJsonProvider
> > 2>/dev/null
> >
> >
> >- Further up in the stack trace I see the error happens when
> initiating
> >the org.apache.hadoop.yarn.util.timeline.TimelineUtils class.  I
> tried
> >setting "yarn.timeline-service.enabled" in Ambari to false and then I
> > get
> >this error:
> >
> > Unable to parse
> > '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a
> > URI, check the setting for mapreduce.application.framework.path
> >
> >
> >- I've tried adding different hadoop, hbase, yarn and mapreduce Maven
> >dependencies without any success
> >   - hadoop-yarn-client
> >   - hadoop-yarn-common
> >   - hadoop-mapreduce-client-core
> >   - hadoop-yarn-server-common
> >   - hadoop-yarn-api
> >   - hbase-server
> >
> > I will keep exploring other possible solutions.  Let me know if anyone
> has
> > any ideas.
> >
> > On Mon, May 7, 2018 at 9:02 AM, Otto Fowler 
> > wrote:
> >
> > > I can imagine a new generic service(s) capability whose job ( pun
> > intended
> > 

Streaming Machine Learning use case

[Ali Nazemian]

Hi all,

I was wondering if someone has used Metron with any streaming ML framework
such as SAMOA? I know that Metron provides Machine Learning separately via
MAAS. However, it is hard to manage it from operational perspective
especially if we want to have a pretty dynamic and evolving model. SAMOA
seems to be a very slow project (or maybe even dead). However, it looks
very close from the integration point of view with Metron, so I wanted to
see if anyone had tried SAMOA in practice and especially with Metron use
cases.

Regards,
Ali