Re: [DISCUSS] Metron debug info tool

[Justin Leet]

First off, this is super nice, and a great way to let us be able to debug
and help others debug quickly, easily, and hopefully more consistently.

I super briefly glanced at at it, so these might already be there, but I'd
like to be able to filter what I get back, e.g. if I give the options for
Storm and Metron, I'd like to limit to just those. Nothing complicated, but
something quick and simple.

Hand in hand with that, I'd like the option to print to screen (maybe just
for the non-config stuff or just print out the relevant filenames?).  At
that point, it'd be really easy to grep or otherwise search through
things.  Tarball is nice, especially when passing things off to someone
else, or when you need to dig through a lot of larger config files, but I
suspect a lot of use cases will be "Hey, real quick what's going on?"

Other than that, does anyone have any thoughts on putting something like
this into the management UI (for the non-Ambari managed stuff)?  That seems
like it would be the natural place to get that stuff, keep it up to date,
and even build in an export if we wanted to.  Would make it a lot easier
for end users to be able to get a quick view into what's going on, and
could let us build in some slightly better filtering and search
capabilities.



On Wed, Apr 11, 2018 at 12:10 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Hey guys,
>
> I wanted to bring attention to a tool I created for gathering cluster
> details for debugging purposes. There are a number of locations that
> properties get materialized, e.g. from Ambari -> properties file -> flux ->
> Storm, which means a lot of hunting to guarantee that the changes you've
> made are percolating correctly. Furthermore, it's generally useful to get a
> sense of how your cluster is configured by gathering all of that info in
> one place. I created a Python tool that does just that, and bundles up the
> results in a tarball. Here is an overview of the artifacts I'm gathering -
> you can see what commands are being used by looking at the script.
>
> Ambari
> full cluster config detail
>
> Storm
> cluster summary
> cluster configuration
> topology summary (enrichments and indexing)
> topology status summary (enrichments and indexing)
>
> Kafka
> broker info
> topics list
> topic details (enrichments and indexing)
>
> Metron
> local file system configuration files
> zookeeper configuration
> flux files
> lib directory file listing
> rpm listing
>
> Hadoop
> version info
>
> *** Are there any features/details you'd like to see added to this? Any
> concerns or suggestions? ***
>
> I am  also planning to add log file support along with md5sum of the jar
> files deployed in Metron's lib directory.
>
> https://github.com/apache/metron/pull/988/files#diff-
> 0eddfa8f1dd67247e0803e405497b6e2
>
> Cheers,
> Mike Miklavcic
>

[DISCUSS] Metron debug info tool

[Michael Miklavcic]

Hey guys,

I wanted to bring attention to a tool I created for gathering cluster
details for debugging purposes. There are a number of locations that
properties get materialized, e.g. from Ambari -> properties file -> flux ->
Storm, which means a lot of hunting to guarantee that the changes you've
made are percolating correctly. Furthermore, it's generally useful to get a
sense of how your cluster is configured by gathering all of that info in
one place. I created a Python tool that does just that, and bundles up the
results in a tarball. Here is an overview of the artifacts I'm gathering -
you can see what commands are being used by looking at the script.

Ambari
full cluster config detail

Storm
cluster summary
cluster configuration
topology summary (enrichments and indexing)
topology status summary (enrichments and indexing)

Kafka
broker info
topics list
topic details (enrichments and indexing)

Metron
local file system configuration files
zookeeper configuration
flux files
lib directory file listing
rpm listing

Hadoop
version info

*** Are there any features/details you'd like to see added to this? Any
concerns or suggestions? ***

I am  also planning to add log file support along with md5sum of the jar
files deployed in Metron's lib directory.

https://github.com/apache/metron/pull/988/files#diff-0eddfa8f1dd67247e0803e405497b6e2

Cheers,
Mike Miklavcic

Re: [DISCUSS] Metron debug info tool

[Michael Miklavcic]

Comments below

On Wed, Apr 11, 2018, 10:59 AM Justin Leet  wrote:

> First off, this is super nice, and a great way to let us be able to debug
> and help others debug quickly, easily, and hopefully more consistently.
>
> I super briefly glanced at at it, so these might already be there, but I'd
> like to be able to filter what I get back, e.g. if I give the options for
> Storm and Metron, I'd like to limit to just those. Nothing complicated, but
> something quick and simple.
>

I had considered that too - I like it. Adding to the list.

>
> Hand in hand with that, I'd like the option to print to screen (maybe just
> for the non-config stuff or just print out the relevant filenames?).  At
> that point, it'd be really easy to grep or otherwise search through
> things.  Tarball is nice, especially when passing things off to someone
> else, or when you need to dig through a lot of larger config files, but I
> suspect a lot of use cases will be "Hey, real quick what's going on?"
>

Another good suggestion. Currently, the files get landed in a local
directory that are tarred up. We could also do a simple dump, similar to
how the zk config tool works.

>
> Other than that, does anyone have any thoughts on putting something like
> this into the management UI (for the non-Ambari managed stuff)?  That seems
> like it would be the natural place to get that stuff, keep it up to date,
> and even build in an export if we wanted to.  Would make it a lot easier
> for end users to be able to get a quick view into what's going on, and
> could let us build in some slightly better filtering and search
> capabilities.
>
>
>
> On Wed, Apr 11, 2018 at 12:10 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Hey guys,
> >
> > I wanted to bring attention to a tool I created for gathering cluster
> > details for debugging purposes. There are a number of locations that
> > properties get materialized, e.g. from Ambari -> properties file -> flux
> ->
> > Storm, which means a lot of hunting to guarantee that the changes you've
> > made are percolating correctly. Furthermore, it's generally useful to
> get a
> > sense of how your cluster is configured by gathering all of that info in
> > one place. I created a Python tool that does just that, and bundles up
> the
> > results in a tarball. Here is an overview of the artifacts I'm gathering
> -
> > you can see what commands are being used by looking at the script.
> >
> > Ambari
> > full cluster config detail
> >
> > Storm
> > cluster summary
> > cluster configuration
> > topology summary (enrichments and indexing)
> > topology status summary (enrichments and indexing)
> >
> > Kafka
> > broker info
> > topics list
> > topic details (enrichments and indexing)
> >
> > Metron
> > local file system configuration files
> > zookeeper configuration
> > flux files
> > lib directory file listing
> > rpm listing
> >
> > Hadoop
> > version info
> >
> > *** Are there any features/details you'd like to see added to this? Any
> > concerns or suggestions? ***
> >
> > I am  also planning to add log file support along with md5sum of the jar
> > files deployed in Metron's lib directory.
> >
> > https://github.com/apache/metron/pull/988/files#diff-
> > 0eddfa8f1dd67247e0803e405497b6e2
> >
> > Cheers,
> > Mike Miklavcic
> >
>

Re: [DISCUSS] Metron debug info tool

[Nick Allen]

I think this is super helpful, Mike.


> Other than that, does anyone have any thoughts on putting something like this
into the management UI (for the non-Ambari managed stuff)?  That seems like
it would be the natural place to get that stuff...

I agree this would be a great feature to add to a UI.

My first thought was that this would be a good addition to Ambari.  I can't
really think of compelling justification to go Ambari or the Mgmt UI
though.  Either would work to make Mike's tool more accessible.





On Wed, Apr 11, 2018 at 12:59 PM, Justin Leet  wrote:

> First off, this is super nice, and a great way to let us be able to debug
> and help others debug quickly, easily, and hopefully more consistently.
>
> I super briefly glanced at at it, so these might already be there, but I'd
> like to be able to filter what I get back, e.g. if I give the options for
> Storm and Metron, I'd like to limit to just those. Nothing complicated, but
> something quick and simple.
>
> Hand in hand with that, I'd like the option to print to screen (maybe just
> for the non-config stuff or just print out the relevant filenames?).  At
> that point, it'd be really easy to grep or otherwise search through
> things.  Tarball is nice, especially when passing things off to someone
> else, or when you need to dig through a lot of larger config files, but I
> suspect a lot of use cases will be "Hey, real quick what's going on?"
>
> Other than that, does anyone have any thoughts on putting something like
> this into the management UI (for the non-Ambari managed stuff)?  That seems
> like it would be the natural place to get that stuff, keep it up to date,
> and even build in an export if we wanted to.  Would make it a lot easier
> for end users to be able to get a quick view into what's going on, and
> could let us build in some slightly better filtering and search
> capabilities.
>
>
>
> On Wed, Apr 11, 2018 at 12:10 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Hey guys,
> >
> > I wanted to bring attention to a tool I created for gathering cluster
> > details for debugging purposes. There are a number of locations that
> > properties get materialized, e.g. from Ambari -> properties file -> flux
> ->
> > Storm, which means a lot of hunting to guarantee that the changes you've
> > made are percolating correctly. Furthermore, it's generally useful to
> get a
> > sense of how your cluster is configured by gathering all of that info in
> > one place. I created a Python tool that does just that, and bundles up
> the
> > results in a tarball. Here is an overview of the artifacts I'm gathering
> -
> > you can see what commands are being used by looking at the script.
> >
> > Ambari
> > full cluster config detail
> >
> > Storm
> > cluster summary
> > cluster configuration
> > topology summary (enrichments and indexing)
> > topology status summary (enrichments and indexing)
> >
> > Kafka
> > broker info
> > topics list
> > topic details (enrichments and indexing)
> >
> > Metron
> > local file system configuration files
> > zookeeper configuration
> > flux files
> > lib directory file listing
> > rpm listing
> >
> > Hadoop
> > version info
> >
> > *** Are there any features/details you'd like to see added to this? Any
> > concerns or suggestions? ***
> >
> > I am  also planning to add log file support along with md5sum of the jar
> > files deployed in Metron's lib directory.
> >
> > https://github.com/apache/metron/pull/988/files#diff-
> > 0eddfa8f1dd67247e0803e405497b6e2
> >
> > Cheers,
> > Mike Miklavcic
> >
>

Re: [DISCUSS] Metron debug info tool

[Michael Miklavcic]

Agree with you both - one reason Ambari might be preferable is that there
are config variables we can access more easily from Ambari, kind of like
what we use in the MPacks. I haven't looked at what we have in the
management UI but I think that's also a reasonable option.

On Wed, Apr 11, 2018, 11:56 AM Nick Allen  wrote:

> I think this is super helpful, Mike.
>
>
> > Other than that, does anyone have any thoughts on putting something like
> this
> into the management UI (for the non-Ambari managed stuff)?  That seems like
> it would be the natural place to get that stuff...
>
> I agree this would be a great feature to add to a UI.
>
> My first thought was that this would be a good addition to Ambari.  I can't
> really think of compelling justification to go Ambari or the Mgmt UI
> though.  Either would work to make Mike's tool more accessible.
>
>
>
>
>
> On Wed, Apr 11, 2018 at 12:59 PM, Justin Leet 
> wrote:
>
> > First off, this is super nice, and a great way to let us be able to debug
> > and help others debug quickly, easily, and hopefully more consistently.
> >
> > I super briefly glanced at at it, so these might already be there, but
> I'd
> > like to be able to filter what I get back, e.g. if I give the options for
> > Storm and Metron, I'd like to limit to just those. Nothing complicated,
> but
> > something quick and simple.
> >
> > Hand in hand with that, I'd like the option to print to screen (maybe
> just
> > for the non-config stuff or just print out the relevant filenames?).  At
> > that point, it'd be really easy to grep or otherwise search through
> > things.  Tarball is nice, especially when passing things off to someone
> > else, or when you need to dig through a lot of larger config files, but I
> > suspect a lot of use cases will be "Hey, real quick what's going on?"
> >
> > Other than that, does anyone have any thoughts on putting something like
> > this into the management UI (for the non-Ambari managed stuff)?  That
> seems
> > like it would be the natural place to get that stuff, keep it up to date,
> > and even build in an export if we wanted to.  Would make it a lot easier
> > for end users to be able to get a quick view into what's going on, and
> > could let us build in some slightly better filtering and search
> > capabilities.
> >
> >
> >
> > On Wed, Apr 11, 2018 at 12:10 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Hey guys,
> > >
> > > I wanted to bring attention to a tool I created for gathering cluster
> > > details for debugging purposes. There are a number of locations that
> > > properties get materialized, e.g. from Ambari -> properties file ->
> flux
> > ->
> > > Storm, which means a lot of hunting to guarantee that the changes
> you've
> > > made are percolating correctly. Furthermore, it's generally useful to
> > get a
> > > sense of how your cluster is configured by gathering all of that info
> in
> > > one place. I created a Python tool that does just that, and bundles up
> > the
> > > results in a tarball. Here is an overview of the artifacts I'm
> gathering
> > -
> > > you can see what commands are being used by looking at the script.
> > >
> > > Ambari
> > > full cluster config detail
> > >
> > > Storm
> > > cluster summary
> > > cluster configuration
> > > topology summary (enrichments and indexing)
> > > topology status summary (enrichments and indexing)
> > >
> > > Kafka
> > > broker info
> > > topics list
> > > topic details (enrichments and indexing)
> > >
> > > Metron
> > > local file system configuration files
> > > zookeeper configuration
> > > flux files
> > > lib directory file listing
> > > rpm listing
> > >
> > > Hadoop
> > > version info
> > >
> > > *** Are there any features/details you'd like to see added to this? Any
> > > concerns or suggestions? ***
> > >
> > > I am  also planning to add log file support along with md5sum of the
> jar
> > > files deployed in Metron's lib directory.
> > >
> > > https://github.com/apache/metron/pull/988/files#diff-
> > > 0eddfa8f1dd67247e0803e405497b6e2
> > >
> > > Cheers,
> > > Mike Miklavcic
> > >
> >
>

Another intermittant build failure?

[Otto Fowler]

I had a PR build fail with an issue with the Zookeeper cache.

https://travis-ci.org/apache/metron/builds/365122993


Failed tests:
  
ZKConfigurationsCacheIntegrationTest.validateUpdate:230->lambda$validateUpdate$9:230
expected:<{hdfs={index=yaf, batchSize=1, enabled=true},
elasticsearch={index=yaf, batchSize=25, batchTimeout=7,
enabled=false}, solr={index=yaf, batchSize=5, enabled=false}}> but
was:<{}>


It passed when I ran it again.

Re: Another intermittant build failure?

[Casey Stella]

I have not personally seen that one yet, but I will not deny that it
exists.  It could be very intermittent or triggered under load in travis
too.  Either way, we should probably investigate and fix.

On Wed, Apr 11, 2018 at 3:57 PM Otto Fowler  wrote:

> I had a PR build fail with an issue with the Zookeeper cache.
>
> https://travis-ci.org/apache/metron/builds/365122993
>
>
> Failed tests:
>
> ZKConfigurationsCacheIntegrationTest.validateUpdate:230->lambda$validateUpdate$9:230
> expected:<{hdfs={index=yaf, batchSize=1, enabled=true},
> elasticsearch={index=yaf, batchSize=25, batchTimeout=7,
> enabled=false}, solr={index=yaf, batchSize=5, enabled=false}}> but
> was:<{}>
>
>
> It passed when I ran it again.
>