Re: [DISCUSS] Metron Management UI

[Nick Allen]

I think its important to keep moving toward a Metron where I can introduce
new telemetry, add new enrichments, tweak the triage process, iteratively,
on-the-fly, and safely in a single, production environment.

If a user wants to have a traditional test-to-production change management
process, they can, but it is not necessary.  I think the base that we have
allows us to do this.

In that regard, I do like the idea, Otto.


On Sat, Feb 18, 2017 at 8:51 AM, Otto Fowler 
wrote:

> I agree with this.  We could really do something cool here if we have
> multiple training topologies, like steps
> that fed the ui.
>
> Step one would be kafka to parser with output to some tmp area.  The ui
> would be used to train the parser/stellar configs
> Step two would add on to enrichments to temp.  The ui would be used to
> train the enrichment configurations
> etc
> etc.
>
> Until it was all set and you were happy then the source was ‘activated’
> with the real topologies.
>
>
>
> On February 18, 2017 at 07:38:17, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
>
> I don't expect people to push data through Metron until they have an
> expectation that it will be properly parsed and handled, otherwise you'll
> flood the cluster with failures and noise.
>
> Here's what's in my head. A user grabs an upstream log sample (from their
> environment), uses the UI to figure out how to parse it, adjusts Metron
> accordingly (make Kafka topic, apply stellar functions, tweak enrich layer,
> triage, optionally index templates) then heads back out of Metron and gives
> it some juice by either just turning on the Kafka producer or doing a spot
> check first (examine in UI things are working properly) then turning on the
> producer (nifi, etc.).
>
> Jon
>
> On Sat, Feb 18, 2017, 5:42 AM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> > At the moment the UI relies on Kafka topics already existing, and having
> > data flowing into them to pull samples for the parser config panel.
> > However, you can paste alternative samples if kafka isn’t already setup.
> As
> > such, the assumption is that the topics would already have data flowing
> > from something like nifi, or some other producer before you got round to
> > configuring a sensor in this UI. Would that work, or would you expect to
> be
> > creating the parser config before the data was flowing into kafka? In the
> > later case, we could probably rely on copy-pasted samples to design the
> > parser, and then auto-create topics if not already existing on parser
> save.
> >
> > Simon
> >
> >
> > > On 17 Feb 2017, at 23:42, zeo...@gmail.com  wrote:
> > >
> > > Well, I think that a UI that does what you show is a huge step forward
> > and
> > > I definitely wouldn't want to stop it from getting into the code
> because
> > of
> > > any of the additional features that we are talking about here.
> > >
> > > That said, I really would like to see the Kafka topic creation as a
> part
> > of
> > > the UI because without it, data can't move. Templates would be a second
> > > priority that I see as important, but depending on how Otto's thread
> goes
> > > how that could would may change and I wouldn't push to have that in a
> MVP
> > > UI.
> > >
> > > Aside from that, I still agree with my previous comments regarding nice
> > to
> > > have features, but definitely not needed for a first run.
> > >
> > > Jon
> > >
> > > On Fri, Feb 17, 2017, 1:24 PM Houshang Livian  >
> > > wrote:
> > >
> > >> Hi Dima and Jon,
> > >>
> > >> Do you think the UI:
> > >>
> > >> http://imgur.com/a/QAQyu?
> > >>
> > >>
> > >> ...Is a reasonable place to start as it is?… or do we need to have the
> > >> Elastic Search Templates right away?
> > >> Are the templates a MUST HAVE in order to add this UI to Metron?
> > >>
> > >> As Simon said, I think we can file a JIRA on the templates and start
> > >> working on what that looks like.
> > >>
> > >> What are your thoughts?
> > >>
> > >>
> > >>
> > >> On 2/17/17, 10:10 AM, "Simon Elliston Ball" <
> > si...@simonellistonball.com>
> > >> wrote:
> > >>
> > >>> Seems like the UI discussion has fallen off the inboxes.
> > >>>
> > >>> We should definitely open up a separate Jira and discussion on the
> best
> > >> way to process elastic search template management. This feels like a
> > non-UI
> > >> question to me, since it would apply equally to any other parser
> config
> > >> method. I’ll start a separate discuss for that.
> > >>>
> > >>> In the meantime it would be great to hear more from the community on
> > the
> > >> UI at http://imgur.com/a/QAQyu?
> > >>>
> > >>> Is there anything you see missing around sensor config here?
> > >>> Does it look useful?
> > >>> Does the flow match the way you would currently think about setting
> up
> > >> sensors?
> > >>>
> > >>> It would be great to get some thoughts.
> > >>>
> > >>> Simon
> > >>>
> > >>>
> >  On 1 Feb 2017, at 12:07, Simon Elliston Ball <

Re: [DISCUSS] Metron Management UI

[James Sirota]

Another thing we can use for this is Stellar shell.  We can pull files from 
kafka (initially one at a time and later in bulk), apply parser, and see a 
result (or results).  If there are any failures you repeat until you get 
everything parsed.  We can then do the same with the enrichmens (that 
capability actually already exists).  

Thanks,
James

18.02.2017, 06:51, "Otto Fowler" :
> I agree with this. We could really do something cool here if we have
> multiple training topologies, like steps
> that fed the ui.
>
> Step one would be kafka to parser with output to some tmp area. The ui
> would be used to train the parser/stellar configs
> Step two would add on to enrichments to temp. The ui would be used to
> train the enrichment configurations
> etc
> etc.
>
> Until it was all set and you were happy then the source was ‘activated’
> with the real topologies.
>
> On February 18, 2017 at 07:38:17, zeo...@gmail.com (zeo...@gmail.com) wrote:
>
> I don't expect people to push data through Metron until they have an
> expectation that it will be properly parsed and handled, otherwise you'll
> flood the cluster with failures and noise.
>
> Here's what's in my head. A user grabs an upstream log sample (from their
> environment), uses the UI to figure out how to parse it, adjusts Metron
> accordingly (make Kafka topic, apply stellar functions, tweak enrich layer,
> triage, optionally index templates) then heads back out of Metron and gives
> it some juice by either just turning on the Kafka producer or doing a spot
> check first (examine in UI things are working properly) then turning on the
> producer (nifi, etc.).
>
> Jon
>
> On Sat, Feb 18, 2017, 5:42 AM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
>>  At the moment the UI relies on Kafka topics already existing, and having
>>  data flowing into them to pull samples for the parser config panel.
>>  However, you can paste alternative samples if kafka isn’t already setup.
>
> As
>>  such, the assumption is that the topics would already have data flowing
>>  from something like nifi, or some other producer before you got round to
>>  configuring a sensor in this UI. Would that work, or would you expect to
>
> be
>>  creating the parser config before the data was flowing into kafka? In the
>>  later case, we could probably rely on copy-pasted samples to design the
>>  parser, and then auto-create topics if not already existing on parser
>
> save.
>>  Simon
>>
>>  > On 17 Feb 2017, at 23:42, zeo...@gmail.com  wrote:
>>  >
>>  > Well, I think that a UI that does what you show is a huge step forward
>>  and
>>  > I definitely wouldn't want to stop it from getting into the code
>
> because
>>  of
>>  > any of the additional features that we are talking about here.
>>  >
>>  > That said, I really would like to see the Kafka topic creation as a
>
> part
>>  of
>>  > the UI because without it, data can't move. Templates would be a second
>>  > priority that I see as important, but depending on how Otto's thread
>
> goes
>>  > how that could would may change and I wouldn't push to have that in a
>
> MVP
>>  > UI.
>>  >
>>  > Aside from that, I still agree with my previous comments regarding nice
>>  to
>>  > have features, but definitely not needed for a first run.
>>  >
>>  > Jon
>>  >
>>  > On Fri, Feb 17, 2017, 1:24 PM Houshang Livian 
>>  > wrote:
>>  >
>>  >> Hi Dima and Jon,
>>  >>
>>  >> Do you think the UI:
>>  >>
>>  >> http://imgur.com/a/QAQyu?
>>  >>
>>  >>
>>  >> ...Is a reasonable place to start as it is?… or do we need to have the
>>  >> Elastic Search Templates right away?
>>  >> Are the templates a MUST HAVE in order to add this UI to Metron?
>>  >>
>>  >> As Simon said, I think we can file a JIRA on the templates and start
>>  >> working on what that looks like.
>>  >>
>>  >> What are your thoughts?
>>  >>
>>  >>
>>  >>
>>  >> On 2/17/17, 10:10 AM, "Simon Elliston Ball" <
>>  si...@simonellistonball.com>
>>  >> wrote:
>>  >>
>>  >>> Seems like the UI discussion has fallen off the inboxes.
>>  >>>
>>  >>> We should definitely open up a separate Jira and discussion on the
>
> best
>>  >> way to process elastic search template management. This feels like a
>>  non-UI
>>  >> question to me, since it would apply equally to any other parser
>
> config
>>  >> method. I’ll start a separate discuss for that.
>>  >>>
>>  >>> In the meantime it would be great to hear more from the community on
>>  the
>>  >> UI at http://imgur.com/a/QAQyu?
>>  >>>
>>  >>> Is there anything you see missing around sensor config here?
>>  >>> Does it look useful?
>>  >>> Does the flow match the way you would currently think about setting
>
> up
>>  >> sensors?
>>  >>>
>>  >>> It would be great to get some thoughts.
>>  >>>
>>  >>> Simon
>>  >>>
>>  >>>
>>   On 1 Feb 2017, at 12:07, Simon Elliston Ball <
>>  >> si...@simonellistonball.com> wrote:
>>  
>>   The MPack creates elastic 

Re: [DISCUSS] Metron Management UI

[Otto Fowler]

I agree with this.  We could really do something cool here if we have
multiple training topologies, like steps
that fed the ui.

Step one would be kafka to parser with output to some tmp area.  The ui
would be used to train the parser/stellar configs
Step two would add on to enrichments to temp.  The ui would be used to
train the enrichment configurations
etc
etc.

Until it was all set and you were happy then the source was ‘activated’
with the real topologies.



On February 18, 2017 at 07:38:17, zeo...@gmail.com (zeo...@gmail.com) wrote:

I don't expect people to push data through Metron until they have an
expectation that it will be properly parsed and handled, otherwise you'll
flood the cluster with failures and noise.

Here's what's in my head. A user grabs an upstream log sample (from their
environment), uses the UI to figure out how to parse it, adjusts Metron
accordingly (make Kafka topic, apply stellar functions, tweak enrich layer,
triage, optionally index templates) then heads back out of Metron and gives
it some juice by either just turning on the Kafka producer or doing a spot
check first (examine in UI things are working properly) then turning on the
producer (nifi, etc.).

Jon

On Sat, Feb 18, 2017, 5:42 AM Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> At the moment the UI relies on Kafka topics already existing, and having
> data flowing into them to pull samples for the parser config panel.
> However, you can paste alternative samples if kafka isn’t already setup.
As
> such, the assumption is that the topics would already have data flowing
> from something like nifi, or some other producer before you got round to
> configuring a sensor in this UI. Would that work, or would you expect to
be
> creating the parser config before the data was flowing into kafka? In the
> later case, we could probably rely on copy-pasted samples to design the
> parser, and then auto-create topics if not already existing on parser
save.
>
> Simon
>
>
> > On 17 Feb 2017, at 23:42, zeo...@gmail.com  wrote:
> >
> > Well, I think that a UI that does what you show is a huge step forward
> and
> > I definitely wouldn't want to stop it from getting into the code
because
> of
> > any of the additional features that we are talking about here.
> >
> > That said, I really would like to see the Kafka topic creation as a
part
> of
> > the UI because without it, data can't move. Templates would be a second
> > priority that I see as important, but depending on how Otto's thread
goes
> > how that could would may change and I wouldn't push to have that in a
MVP
> > UI.
> >
> > Aside from that, I still agree with my previous comments regarding nice
> to
> > have features, but definitely not needed for a first run.
> >
> > Jon
> >
> > On Fri, Feb 17, 2017, 1:24 PM Houshang Livian 
> > wrote:
> >
> >> Hi Dima and Jon,
> >>
> >> Do you think the UI:
> >>
> >> http://imgur.com/a/QAQyu?
> >>
> >>
> >> ...Is a reasonable place to start as it is?… or do we need to have the
> >> Elastic Search Templates right away?
> >> Are the templates a MUST HAVE in order to add this UI to Metron?
> >>
> >> As Simon said, I think we can file a JIRA on the templates and start
> >> working on what that looks like.
> >>
> >> What are your thoughts?
> >>
> >>
> >>
> >> On 2/17/17, 10:10 AM, "Simon Elliston Ball" <
> si...@simonellistonball.com>
> >> wrote:
> >>
> >>> Seems like the UI discussion has fallen off the inboxes.
> >>>
> >>> We should definitely open up a separate Jira and discussion on the
best
> >> way to process elastic search template management. This feels like a
> non-UI
> >> question to me, since it would apply equally to any other parser
config
> >> method. I’ll start a separate discuss for that.
> >>>
> >>> In the meantime it would be great to hear more from the community on
> the
> >> UI at http://imgur.com/a/QAQyu?
> >>>
> >>> Is there anything you see missing around sensor config here?
> >>> Does it look useful?
> >>> Does the flow match the way you would currently think about setting
up
> >> sensors?
> >>>
> >>> It would be great to get some thoughts.
> >>>
> >>> Simon
> >>>
> >>>
>  On 1 Feb 2017, at 12:07, Simon Elliston Ball <
> >> si...@simonellistonball.com> wrote:
> 
>  The MPack creates elastic templates for the standard parsers
included
> >> in the pack.
> 
>  However, when creating new sensors we should be able to infer
Elastic
> >> templates from the schema. There are certain fields that we need to
mark
> >> types for to generate those templates (e.g. date / time fields). This
> goes
> >> back to the debate we had a while ago about, for example, typing in
the
> >> grok parser, but applies for others too. I suspect that since we
should
> be
> >> able to infer all the types needed for templates, this is more an
> excellent
> >> feature enhancement for the REST api layer, unless anyone can think of
a
> >> reason we would need explicit type 

Re: [DISCUSS] Metron Management UI

[zeo...@gmail.com]

I don't expect people to push data through Metron until they have an
expectation that it will be properly parsed and handled, otherwise you'll
flood the cluster with failures and noise.

Here's what's in my head.  A user grabs an upstream log sample (from their
environment), uses the UI to figure out how to parse it, adjusts Metron
accordingly (make Kafka topic, apply stellar functions, tweak enrich layer,
triage, optionally index templates) then heads back out of Metron and gives
it some juice by either just turning on the Kafka producer or doing a spot
check first (examine in UI things are working properly) then turning on the
producer (nifi, etc.).

Jon

On Sat, Feb 18, 2017, 5:42 AM Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> At the moment the UI relies on Kafka topics already existing, and having
> data flowing into them to pull samples for the parser config panel.
> However, you can paste alternative samples if kafka isn’t already setup. As
> such, the assumption is that the topics would already have data flowing
> from something like nifi, or some other producer before you got round to
> configuring a sensor in this UI. Would that work, or would you expect to be
> creating the parser config before the data was flowing into kafka? In the
> later case, we could probably rely on copy-pasted samples to design the
> parser, and then auto-create topics if not already existing on parser save.
>
> Simon
>
>
> > On 17 Feb 2017, at 23:42, zeo...@gmail.com  wrote:
> >
> > Well, I think that a UI that does what you show is a huge step forward
> and
> > I definitely wouldn't want to stop it from getting into the code because
> of
> > any of the additional features that we are talking about here.
> >
> > That said, I really would like to see the Kafka topic creation as a part
> of
> > the UI because without it, data can't move.  Templates would be a second
> > priority that I see as important, but depending on how Otto's thread goes
> > how that could would may change and I wouldn't push to have that in a MVP
> > UI.
> >
> > Aside from that, I still agree with my previous comments regarding nice
> to
> > have features, but definitely not needed for a first run.
> >
> > Jon
> >
> > On Fri, Feb 17, 2017, 1:24 PM Houshang Livian 
> > wrote:
> >
> >> Hi Dima and Jon,
> >>
> >> Do you think the UI:
> >>
> >> http://imgur.com/a/QAQyu?
> >>
> >>
> >> ...Is a reasonable place to start as it is?… or do we need to have the
> >> Elastic Search Templates right away?
> >> Are the templates a MUST HAVE in order to add this UI to Metron?
> >>
> >> As Simon said, I think we can file a JIRA on the templates and start
> >> working on what that looks like.
> >>
> >> What are your thoughts?
> >>
> >>
> >>
> >> On 2/17/17, 10:10 AM, "Simon Elliston Ball" <
> si...@simonellistonball.com>
> >> wrote:
> >>
> >>> Seems like the UI discussion has fallen off the inboxes.
> >>>
> >>> We should definitely open up a separate Jira and discussion on the best
> >> way to process elastic search template management. This feels like a
> non-UI
> >> question to me, since it would apply equally to any other parser config
> >> method. I’ll start a separate discuss for that.
> >>>
> >>> In the meantime it would be great to hear more from the community on
> the
> >> UI at http://imgur.com/a/QAQyu?
> >>>
> >>> Is there anything you see missing around sensor config here?
> >>> Does it look useful?
> >>> Does the flow match the way you would currently think about setting up
> >> sensors?
> >>>
> >>> It would be great to get some thoughts.
> >>>
> >>> Simon
> >>>
> >>>
>  On 1 Feb 2017, at 12:07, Simon Elliston Ball <
> >> si...@simonellistonball.com> wrote:
> 
>  The MPack creates elastic templates for the standard parsers included
> >> in the pack.
> 
>  However, when creating new sensors we should be able to infer Elastic
> >> templates from the schema. There are certain fields that we need to mark
> >> types for to generate those templates (e.g. date / time fields). This
> goes
> >> back to the debate we had a while ago about, for example, typing in the
> >> grok parser, but applies for others too. I suspect that since we should
> be
> >> able to infer all the types needed for templates, this is more an
> excellent
> >> feature enhancement for the REST api layer, unless anyone can think of a
> >> reason we would need explicit type definition in the
> 
> 
> 
> > On 1 Feb 2017, at 19:54, Nick Allen  wrote:
> >
> > The Index templates are currently handled in the Ambari MPack (unless
> >> you
> > are suggesting some additional functionality beyond what the Ambari
> >> MPack
> > gives us today.)
> >
> > I think there is always going to be a fuzzy, gray line between what
> the
> > Ambari MPack should do and what the Management UI should do.  I think
> >> it
> > works fine in Ambari today, but I could be 

Re: [DISCUSS] Metron Management UI

[Simon Elliston Ball]

At the moment the UI relies on Kafka topics already existing, and having data 
flowing into them to pull samples for the parser config panel. However, you can 
paste alternative samples if kafka isn’t already setup. As such, the assumption 
is that the topics would already have data flowing from something like nifi, or 
some other producer before you got round to configuring a sensor in this UI. 
Would that work, or would you expect to be creating the parser config before 
the data was flowing into kafka? In the later case, we could probably rely on 
copy-pasted samples to design the parser, and then auto-create topics if not 
already existing on parser save.

Simon


> On 17 Feb 2017, at 23:42, zeo...@gmail.com  wrote:
> 
> Well, I think that a UI that does what you show is a huge step forward and
> I definitely wouldn't want to stop it from getting into the code because of
> any of the additional features that we are talking about here.
> 
> That said, I really would like to see the Kafka topic creation as a part of
> the UI because without it, data can't move.  Templates would be a second
> priority that I see as important, but depending on how Otto's thread goes
> how that could would may change and I wouldn't push to have that in a MVP
> UI.
> 
> Aside from that, I still agree with my previous comments regarding nice to
> have features, but definitely not needed for a first run.
> 
> Jon
> 
> On Fri, Feb 17, 2017, 1:24 PM Houshang Livian 
> wrote:
> 
>> Hi Dima and Jon,
>> 
>> Do you think the UI:
>> 
>> http://imgur.com/a/QAQyu?
>> 
>> 
>> ...Is a reasonable place to start as it is?… or do we need to have the
>> Elastic Search Templates right away?
>> Are the templates a MUST HAVE in order to add this UI to Metron?
>> 
>> As Simon said, I think we can file a JIRA on the templates and start
>> working on what that looks like.
>> 
>> What are your thoughts?
>> 
>> 
>> 
>> On 2/17/17, 10:10 AM, "Simon Elliston Ball" 
>> wrote:
>> 
>>> Seems like the UI discussion has fallen off the inboxes.
>>> 
>>> We should definitely open up a separate Jira and discussion on the best
>> way to process elastic search template management. This feels like a non-UI
>> question to me, since it would apply equally to any other parser config
>> method. I’ll start a separate discuss for that.
>>> 
>>> In the meantime it would be great to hear more from the community on the
>> UI at http://imgur.com/a/QAQyu?
>>> 
>>> Is there anything you see missing around sensor config here?
>>> Does it look useful?
>>> Does the flow match the way you would currently think about setting up
>> sensors?
>>> 
>>> It would be great to get some thoughts.
>>> 
>>> Simon
>>> 
>>> 
 On 1 Feb 2017, at 12:07, Simon Elliston Ball <
>> si...@simonellistonball.com> wrote:
 
 The MPack creates elastic templates for the standard parsers included
>> in the pack.
 
 However, when creating new sensors we should be able to infer Elastic
>> templates from the schema. There are certain fields that we need to mark
>> types for to generate those templates (e.g. date / time fields). This goes
>> back to the debate we had a while ago about, for example, typing in the
>> grok parser, but applies for others too. I suspect that since we should be
>> able to infer all the types needed for templates, this is more an excellent
>> feature enhancement for the REST api layer, unless anyone can think of a
>> reason we would need explicit type definition in the
 
 
 
> On 1 Feb 2017, at 19:54, Nick Allen  wrote:
> 
> The Index templates are currently handled in the Ambari MPack (unless
>> you
> are suggesting some additional functionality beyond what the Ambari
>> MPack
> gives us today.)
> 
> I think there is always going to be a fuzzy, gray line between what the
> Ambari MPack should do and what the Management UI should do.  I think
>> it
> works fine in Ambari today, but I could be convinced otherwise.
> 
> 
> 
> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov >> 
> wrote:
> 
>> Thank you Houshang,
>> 
>> That looks absolutely great, loved it!
>> Will it make sense to also add ElasticSearch index  templates
>> querying/creation here?
>> 
>> - Dima
>> 
>> On 02/01/2017 02:06 AM, Houshang Livian wrote:
>>> Hello Metron Community,
>>> 
>>> We have constructed a Management Module UI, built on top of
>> METRON-503
>> (REST API) (currently under review). This Module gives users the
>> ability to
>> setup and administer much of the product through the UI.
>>> 
>>> Here are some screens to show you what we are thinking. Please take a
>> look:
>>> http://imgur.com/a/QAQyu?
>>> 
>>> Does this look like a reasonable place to start?
>>> Is there anything that is an absolute MUST have or MUST NOT 

Re: [DISCUSS] Metron Management UI

[zeo...@gmail.com]

Well, I think that a UI that does what you show is a huge step forward and
I definitely wouldn't want to stop it from getting into the code because of
any of the additional features that we are talking about here.

That said, I really would like to see the Kafka topic creation as a part of
the UI because without it, data can't move.  Templates would be a second
priority that I see as important, but depending on how Otto's thread goes
how that could would may change and I wouldn't push to have that in a MVP
UI.

Aside from that, I still agree with my previous comments regarding nice to
have features, but definitely not needed for a first run.

Jon

On Fri, Feb 17, 2017, 1:24 PM Houshang Livian 
wrote:

> Hi Dima and Jon,
>
> Do you think the UI:
>
> http://imgur.com/a/QAQyu?
>
>
> ...Is a reasonable place to start as it is?… or do we need to have the
> Elastic Search Templates right away?
> Are the templates a MUST HAVE in order to add this UI to Metron?
>
> As Simon said, I think we can file a JIRA on the templates and start
> working on what that looks like.
>
> What are your thoughts?
>
>
>
> On 2/17/17, 10:10 AM, "Simon Elliston Ball" 
> wrote:
>
> >Seems like the UI discussion has fallen off the inboxes.
> >
> >We should definitely open up a separate Jira and discussion on the best
> way to process elastic search template management. This feels like a non-UI
> question to me, since it would apply equally to any other parser config
> method. I’ll start a separate discuss for that.
> >
> >In the meantime it would be great to hear more from the community on the
> UI at http://imgur.com/a/QAQyu?
> >
> >Is there anything you see missing around sensor config here?
> >Does it look useful?
> >Does the flow match the way you would currently think about setting up
> sensors?
> >
> >It would be great to get some thoughts.
> >
> >Simon
> >
> >
> >> On 1 Feb 2017, at 12:07, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
> >>
> >> The MPack creates elastic templates for the standard parsers included
> in the pack.
> >>
> >> However, when creating new sensors we should be able to infer Elastic
> templates from the schema. There are certain fields that we need to mark
> types for to generate those templates (e.g. date / time fields). This goes
> back to the debate we had a while ago about, for example, typing in the
> grok parser, but applies for others too. I suspect that since we should be
> able to infer all the types needed for templates, this is more an excellent
> feature enhancement for the REST api layer, unless anyone can think of a
> reason we would need explicit type definition in the
> >>
> >>
> >>
> >>> On 1 Feb 2017, at 19:54, Nick Allen  wrote:
> >>>
> >>> The Index templates are currently handled in the Ambari MPack (unless
> you
> >>> are suggesting some additional functionality beyond what the Ambari
> MPack
> >>> gives us today.)
> >>>
> >>> I think there is always going to be a fuzzy, gray line between what the
> >>> Ambari MPack should do and what the Management UI should do.  I think
> it
> >>> works fine in Ambari today, but I could be convinced otherwise.
> >>>
> >>>
> >>>
> >>> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov  >
> >>> wrote:
> >>>
>  Thank you Houshang,
> 
>  That looks absolutely great, loved it!
>  Will it make sense to also add ElasticSearch index  templates
>  querying/creation here?
> 
>  - Dima
> 
>  On 02/01/2017 02:06 AM, Houshang Livian wrote:
> > Hello Metron Community,
> >
> > We have constructed a Management Module UI, built on top of
> METRON-503
>  (REST API) (currently under review). This Module gives users the
> ability to
>  setup and administer much of the product through the UI.
> >
> > Here are some screens to show you what we are thinking. Please take a
>  look:
> > http://imgur.com/a/QAQyu?
> >
> > Does this look like a reasonable place to start?
> > Is there anything that is an absolute MUST have or MUST NOT have?
> >
> > Houshang Livian
> > Senior User Experience Designer
> > Hortonworks
> >
> > www.hortonworks.com
> > 
> >
> > Mobile: (831) 521-4176
> > hliv...@hortonworks.com
> > 
> >
> >
> 
> 
> >>>
> >>>
> >>> --
> >>> Nick Allen 
> >>
> >
>
-- 

Jon

Sent from my mobile device

Re: [DISCUSS] Metron Management UI

[Houshang Livian]

Hi Dima and Jon,

Do you think the UI:

http://imgur.com/a/QAQyu?


...Is a reasonable place to start as it is?… or do we need to have the Elastic 
Search Templates right away?
Are the templates a MUST HAVE in order to add this UI to Metron?

As Simon said, I think we can file a JIRA on the templates and start working on 
what that looks like.

What are your thoughts?



On 2/17/17, 10:10 AM, "Simon Elliston Ball"  wrote:

>Seems like the UI discussion has fallen off the inboxes. 
>
>We should definitely open up a separate Jira and discussion on the best way to 
>process elastic search template management. This feels like a non-UI question 
>to me, since it would apply equally to any other parser config method. I’ll 
>start a separate discuss for that. 
>
>In the meantime it would be great to hear more from the community on the UI at 
>http://imgur.com/a/QAQyu?
>
>Is there anything you see missing around sensor config here? 
>Does it look useful? 
>Does the flow match the way you would currently think about setting up 
>sensors? 
>
>It would be great to get some thoughts. 
>
>Simon
>
>
>> On 1 Feb 2017, at 12:07, Simon Elliston Ball  
>> wrote:
>> 
>> The MPack creates elastic templates for the standard parsers included in the 
>> pack. 
>> 
>> However, when creating new sensors we should be able to infer Elastic 
>> templates from the schema. There are certain fields that we need to mark 
>> types for to generate those templates (e.g. date / time fields). This goes 
>> back to the debate we had a while ago about, for example, typing in the grok 
>> parser, but applies for others too. I suspect that since we should be able 
>> to infer all the types needed for templates, this is more an excellent 
>> feature enhancement for the REST api layer, unless anyone can think of a 
>> reason we would need explicit type definition in the 
>> 
>> 
>> 
>>> On 1 Feb 2017, at 19:54, Nick Allen  wrote:
>>> 
>>> The Index templates are currently handled in the Ambari MPack (unless you
>>> are suggesting some additional functionality beyond what the Ambari MPack
>>> gives us today.)
>>> 
>>> I think there is always going to be a fuzzy, gray line between what the
>>> Ambari MPack should do and what the Management UI should do.  I think it
>>> works fine in Ambari today, but I could be convinced otherwise.
>>> 
>>> 
>>> 
>>> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov 
>>> wrote:
>>> 
 Thank you Houshang,
 
 That looks absolutely great, loved it!
 Will it make sense to also add ElasticSearch index  templates
 querying/creation here?
 
 - Dima
 
 On 02/01/2017 02:06 AM, Houshang Livian wrote:
> Hello Metron Community,
> 
> We have constructed a Management Module UI, built on top of METRON-503
 (REST API) (currently under review). This Module gives users the ability to
 setup and administer much of the product through the UI.
> 
> Here are some screens to show you what we are thinking. Please take a
 look:
> http://imgur.com/a/QAQyu?
> 
> Does this look like a reasonable place to start?
> Is there anything that is an absolute MUST have or MUST NOT have?
> 
> Houshang Livian
> Senior User Experience Designer
> Hortonworks
> 
> www.hortonworks.com
> 
> 
> Mobile: (831) 521-4176
> hliv...@hortonworks.com
> 
> 
> 
 
 
>>> 
>>> 
>>> -- 
>>> Nick Allen 
>> 
>

Re: [DISCUSS] Metron Management UI

[Simon Elliston Ball]

Seems like the UI discussion has fallen off the inboxes. 

We should definitely open up a separate Jira and discussion on the best way to 
process elastic search template management. This feels like a non-UI question 
to me, since it would apply equally to any other parser config method. I’ll 
start a separate discuss for that. 

In the meantime it would be great to hear more from the community on the UI at 
http://imgur.com/a/QAQyu?

Is there anything you see missing around sensor config here? 
Does it look useful? 
Does the flow match the way you would currently think about setting up sensors? 

It would be great to get some thoughts. 

Simon


> On 1 Feb 2017, at 12:07, Simon Elliston Ball  
> wrote:
> 
> The MPack creates elastic templates for the standard parsers included in the 
> pack. 
> 
> However, when creating new sensors we should be able to infer Elastic 
> templates from the schema. There are certain fields that we need to mark 
> types for to generate those templates (e.g. date / time fields). This goes 
> back to the debate we had a while ago about, for example, typing in the grok 
> parser, but applies for others too. I suspect that since we should be able to 
> infer all the types needed for templates, this is more an excellent feature 
> enhancement for the REST api layer, unless anyone can think of a reason we 
> would need explicit type definition in the 
> 
> 
> 
>> On 1 Feb 2017, at 19:54, Nick Allen  wrote:
>> 
>> The Index templates are currently handled in the Ambari MPack (unless you
>> are suggesting some additional functionality beyond what the Ambari MPack
>> gives us today.)
>> 
>> I think there is always going to be a fuzzy, gray line between what the
>> Ambari MPack should do and what the Management UI should do.  I think it
>> works fine in Ambari today, but I could be convinced otherwise.
>> 
>> 
>> 
>> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov 
>> wrote:
>> 
>>> Thank you Houshang,
>>> 
>>> That looks absolutely great, loved it!
>>> Will it make sense to also add ElasticSearch index  templates
>>> querying/creation here?
>>> 
>>> - Dima
>>> 
>>> On 02/01/2017 02:06 AM, Houshang Livian wrote:
 Hello Metron Community,
 
 We have constructed a Management Module UI, built on top of METRON-503
>>> (REST API) (currently under review). This Module gives users the ability to
>>> setup and administer much of the product through the UI.
 
 Here are some screens to show you what we are thinking. Please take a
>>> look:
 http://imgur.com/a/QAQyu?
 
 Does this look like a reasonable place to start?
 Is there anything that is an absolute MUST have or MUST NOT have?
 
 Houshang Livian
 Senior User Experience Designer
 Hortonworks
 
 www.hortonworks.com
 
 
 Mobile: (831) 521-4176
 hliv...@hortonworks.com
 
 
 
>>> 
>>> 
>> 
>> 
>> -- 
>> Nick Allen 
> 

Re: [DISCUSS] Metron Management UI

[Otto Fowler]

It is not an either or proposition, we can still have management in Ambari
and have the Metron UI call Ambari to perform some tasks as well.  We do
not want to have to re-write ambari functionality.



On February 5, 2017 at 02:27:53, Dima Kovalyov (dima.koval...@sstech.us)
wrote:

Thank you Jon, for accepting and pushing ES template suggestion further,
I do agree.

Houshang, I am not sure if it is MUST, but it is something that makes
sense if you allow user to create new parser. It is for absolutely OK to
just start here.

For me, it makes sense to distinguish that Metron UI will be managed by
Analytics department while Ambari UI may be handled by DevOpvs dep (does
it make sense?). So if we allow Analyst to create/tweak parsers in
Metron UI we should also move ES templates modification feature out of
Ambari UI to Metron UI. If that is so, should Metron service (that
controls topologies like Enrichment and Indexing) be moved to Metron UI?

What about health status monitoring? Considering that plan is replace
monit with Ambari Mpack where new topologies will be monitored?

Last thing, should metron-parsers*.jar be split into separate parsers
jar in case if among 50 parsers we need to update just one and we want
to do that from Metron UI?

- Dima

On 02/01/2017 11:16 PM, zeo...@gmail.com wrote:
> Ok great, thanks for the feedback Ryan. I'm going to try and get around to
> playing with this next week if I can. Currently my productivity machine is
> "in the shop" getting the battery replaced, so I'm playing man down right
> now.
>
> So if we ignore the current API restrictions, here are my MUST haves:
>
> - We should be able to update search/retrieve (i.e. I agree with Dima).
> This means tweaking the ES templates, because when putting together a new
> sensor or modifying an existing one, we will likely have a new field or
> entirely new template.
> - When creating a new sensor, it should create a new kafka topic (if it
> doesn't already - it may, I just haven't played with it yet) with some
very
> simple/dumb default settings (1 partition, default topic size, 0
> replication).
>
> And here are my NICE to haves (eventually):
> - You should have the ability to tune a kafka topic's config from the UI,
> such as setting the topic size, # of partitions, and replication number.
> This is especially useful if you just created the kafka topic in the UI.
> Of course this should have lots of warnings about the issues with
> decreasing these numbers in the future, and maybe a "read more" link to
> some related documentation in the kafka documentation.
>
> - Tune and restart the topologies to add parallelism into storm (things
> like the --parser_p, --parser_num_tasks, --num_workers, etc. options).
> - It would be nice to be able to modify the jobs that purge data from HDFS
> and ES via the UI. Managing the cluster should include setting your data
> retention wants - I think these are still purge cron jobs? For some
> context, I'm thinking about how this could manage METRON-477 (both the
> original ticket as well as Nick's comments for additional features), which
> could be managed in this UI.
> - The ability to generate a mpack that would install the currently
> configured state of the cluster (not including data, of course, but
> focusing on setting new, tweaked defaults to be what is currently
> configured). This would be useful for rebuilds, standing up new clusters,
> or migrating to a different environment. IMO, this is a really slick
> feature because it allows a nice migration from dev > test > prod.
>
>
> Not to get off topic, but I think the ES side of the house could generally
> use some love. For instance, I don't see a reason why every string field
> shouldn't have '"ignore_above": 8191' on it, at least right now. I will
> probably do this as a part of METRON-635, but we should make sure to carry
> that setting forward appropriately (i.e. if we infer ES templates from a
> schema). I could agree with ES templates being inferred when creating new
> parsers/sensors as a nice to have, building off of a must have of being
> able to make/tweak ES templates in the UI/API.
>
>
> Ok, that's a lot of text, I'll end it there. =D
>
>
> Jon
>
> On Wed, Feb 1, 2017 at 3:07 PM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> The MPack creates elastic templates for the standard parsers included in
> the pack.
>
> However, when creating new sensors we should be able to infer Elastic
> templates from the schema. There are certain fields that we need to mark
> types for to generate those templates (e.g. date / time fields). This goes
> back to the debate we had a while ago about, for example, typing in the
> grok parser, but applies for others too. I suspect that since we should be
> able to infer all the types needed for templates, this is more an
excellent
> feature enhancement for the REST api layer, unless anyone can think of a
> reason we would need explicit type definition in the schema config element
> 

Re: [DISCUSS] Metron Management UI

[zeo...@gmail.com]

Thanks Dima.  I think that division does make sense.  We are specifically
talking about the Metron management UI in this conversation (not
necessarily the Analyst or Investigator's day to day console) and so how
this works in use will obviously depend on what roles exist in a company
because they won't always perfectly match Metron personnas.  In my scenario
we will have a hadoop admin team that has very limited access to Metron
directly (I.e. Ambari but not Metron) specifically for sensitivity reasons.

Second, when I say must, I mean must exist for a complete cluster
management experience using the UI, but I agree, it doesn't need to be a
part of the initial merge.  I'm trying to keep a user from needing to
switch between using the UI and CLI to bring a new log source into the
cluster.

Regarding health monitoring, there is an ongoing dev mailing list thread on
changing the current error reporting, which includes the idea of an ES
dashboard for the errors.  Are you suggesting something in addition to
that?  I would be in favor of more health/performance monitoring details,
but I would need to think carefully about what parts of that to happen in
the management UI as opposed to Ambari.

What about a stoplight area (red, yellow, green) in the mgmt UI that links
to Ambari?  Or are you more interested in metrics like cluster
throughput/performance tuning ("health" as in is the cluster is backing up
or not)?  I could see that part being in the mgmt UI, alongside some of the
nice to haves I suggested to modify the related parameters.

Jon

On Sun, Feb 5, 2017, 2:27 AM Dima Kovalyov  wrote:

Thank you Jon, for accepting and pushing ES template suggestion further,
I do agree.

Houshang, I am not sure if it is MUST, but it is something that makes
sense if you allow user to create new parser. It is for absolutely OK to
just start here.

For me, it makes sense to distinguish that Metron UI will be managed by
Analytics department while Ambari UI may be handled by DevOpvs dep (does
it make sense?). So if we allow Analyst to create/tweak parsers in
Metron UI we should also move ES templates modification feature out of
Ambari UI to Metron UI. If that is so, should Metron service (that
controls topologies like Enrichment and Indexing) be moved to Metron UI?

What about health status monitoring? Considering that plan is replace
monit with Ambari Mpack where new topologies will be monitored?

Last thing, should metron-parsers*.jar be split into separate parsers
jar in case if among 50 parsers we need to update just one and we want
to do that from Metron UI?

- Dima

On 02/01/2017 11:16 PM, zeo...@gmail.com wrote:
> Ok great, thanks for the feedback Ryan.  I'm going to try and get around
to
> playing with this next week if I can.  Currently my productivity machine
is
> "in the shop" getting the battery replaced, so I'm playing man down right
> now.
>
> So if we ignore the current API restrictions, here are my MUST haves:
>
>  - We should be able to update search/retrieve (i.e. I agree with Dima).
> This means tweaking the ES templates, because when putting together a new
> sensor or modifying an existing one, we will likely have a new field or
> entirely new template.
>  - When creating a new sensor, it should create a new kafka topic (if it
> doesn't already - it may, I just haven't played with it yet) with some
very
> simple/dumb default settings (1 partition, default topic size, 0
> replication).
>
> And here are my NICE to haves (eventually):
>  - You should have the ability to tune a kafka topic's config from the UI,
> such as setting the topic size, # of partitions, and replication number.
> This is especially useful if you just created the kafka topic in the UI.
> Of course this should have lots of warnings about the issues with
> decreasing these numbers in the future, and maybe a "read more" link to
> some related documentation in the kafka documentation.
>
>  - Tune and restart the topologies to add parallelism into storm (things
> like the --parser_p, --parser_num_tasks, --num_workers, etc. options).
>  - It would be nice to be able to modify the jobs that purge data from
HDFS
> and ES via the UI.  Managing the cluster should include setting your data
> retention wants - I think these are still purge cron jobs?  For some
> context, I'm thinking about how this could manage METRON-477 (both the
> original ticket as well as Nick's comments for additional features), which
> could be managed in this UI.
>  - The ability to generate a mpack that would install the currently
> configured state of the cluster (not including data, of course, but
> focusing on setting new, tweaked defaults to be what is currently
> configured).  This would be useful for rebuilds, standing up new clusters,
> or migrating to a different environment.  IMO, this is a really slick
> feature because it allows a nice migration from dev > test > prod.
>
>
> Not to get off topic, but I think the ES side of the house could 

Re: [DISCUSS] Metron Management UI

[Dima Kovalyov]

Thank you Jon, for accepting and pushing ES template suggestion further,
I do agree.

Houshang, I am not sure if it is MUST, but it is something that makes
sense if you allow user to create new parser. It is for absolutely OK to
just start here.

For me, it makes sense to distinguish that Metron UI will be managed by
Analytics department while Ambari UI may be handled by DevOpvs dep (does
it make sense?). So if we allow Analyst to create/tweak parsers in
Metron UI we should also move ES templates modification feature out of
Ambari UI to Metron UI. If that is so, should Metron service (that
controls topologies like Enrichment and Indexing) be moved to Metron UI?

What about health status monitoring? Considering that plan is replace
monit with Ambari Mpack where new topologies will be monitored?

Last thing, should metron-parsers*.jar be split into separate parsers
jar in case if among 50 parsers we need to update just one and we want
to do that from Metron UI?

- Dima

On 02/01/2017 11:16 PM, zeo...@gmail.com wrote:
> Ok great, thanks for the feedback Ryan.  I'm going to try and get around to
> playing with this next week if I can.  Currently my productivity machine is
> "in the shop" getting the battery replaced, so I'm playing man down right
> now.
>
> So if we ignore the current API restrictions, here are my MUST haves:
>
>  - We should be able to update search/retrieve (i.e. I agree with Dima).
> This means tweaking the ES templates, because when putting together a new
> sensor or modifying an existing one, we will likely have a new field or
> entirely new template.
>  - When creating a new sensor, it should create a new kafka topic (if it
> doesn't already - it may, I just haven't played with it yet) with some very
> simple/dumb default settings (1 partition, default topic size, 0
> replication).
>
> And here are my NICE to haves (eventually):
>  - You should have the ability to tune a kafka topic's config from the UI,
> such as setting the topic size, # of partitions, and replication number.
> This is especially useful if you just created the kafka topic in the UI.
> Of course this should have lots of warnings about the issues with
> decreasing these numbers in the future, and maybe a "read more" link to
> some related documentation in the kafka documentation.
>
>  - Tune and restart the topologies to add parallelism into storm (things
> like the --parser_p, --parser_num_tasks, --num_workers, etc. options).
>  - It would be nice to be able to modify the jobs that purge data from HDFS
> and ES via the UI.  Managing the cluster should include setting your data
> retention wants - I think these are still purge cron jobs?  For some
> context, I'm thinking about how this could manage METRON-477 (both the
> original ticket as well as Nick's comments for additional features), which
> could be managed in this UI.
>  - The ability to generate a mpack that would install the currently
> configured state of the cluster (not including data, of course, but
> focusing on setting new, tweaked defaults to be what is currently
> configured).  This would be useful for rebuilds, standing up new clusters,
> or migrating to a different environment.  IMO, this is a really slick
> feature because it allows a nice migration from dev > test > prod.
>
>
> Not to get off topic, but I think the ES side of the house could generally
> use some love.  For instance, I don't see a reason why every string field
> shouldn't have '"ignore_above": 8191' on it, at least right now.  I will
> probably do this as a part of METRON-635, but we should make sure to carry
> that setting forward appropriately (i.e. if we infer ES templates from a
> schema).  I could agree with ES templates being inferred when creating new
> parsers/sensors as a nice to have, building off of a must have of being
> able to make/tweak ES templates in the UI/API.
>
>
> Ok, that's a lot of text, I'll end it there.  =D
>
>
> Jon
>
> On Wed, Feb 1, 2017 at 3:07 PM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> The MPack creates elastic templates for the standard parsers included in
> the pack.
>
> However, when creating new sensors we should be able to infer Elastic
> templates from the schema. There are certain fields that we need to mark
> types for to generate those templates (e.g. date / time fields). This goes
> back to the debate we had a while ago about, for example, typing in the
> grok parser, but applies for others too. I suspect that since we should be
> able to infer all the types needed for templates, this is more an excellent
> feature enhancement for the REST api layer, unless anyone can think of a
> reason we would need explicit type definition in the schema config element
> of the UI. Thoughts?
>
> Simon
>
>
>> On 1 Feb 2017, at 19:54, Nick Allen  wrote:
>>
>> The Index templates are currently handled in the Ambari MPack (unless you
>> are suggesting some additional functionality beyond what the Ambari MPack
>> gives 

Re: [DISCUSS] Metron Management UI

[Dima Kovalyov]

Thank you Jon, for accepting and pushing ES template suggestion further,
I do agree.

For me, it makes sense to distinguish that Metron UI will be managed by
Analytics department while Ambari UI may be handled by DevOpvs dep. So
if we allow Analyst to create/tweak parsers in Metron UI we should also
move ES templates modification feature out of Ambari UI to Metron UI. If
that is so, should Metron service that controls default topologies (like
Enrichment and Indexing) be moved to Metron UI?

What about health status monitoring? Considering that plan is replace
monit with Ambari Mpack where new topologies will be monitored?

- Dima

On 02/01/2017 11:16 PM, zeo...@gmail.com wrote:
> Ok great, thanks for the feedback Ryan.  I'm going to try and get around to
> playing with this next week if I can.  Currently my productivity machine is
> "in the shop" getting the battery replaced, so I'm playing man down right
> now.
>
> So if we ignore the current API restrictions, here are my MUST haves:
>
>  - We should be able to update search/retrieve (i.e. I agree with Dima).
> This means tweaking the ES templates, because when putting together a new
> sensor or modifying an existing one, we will likely have a new field or
> entirely new template.
>  - When creating a new sensor, it should create a new kafka topic (if it
> doesn't already - it may, I just haven't played with it yet) with some very
> simple/dumb default settings (1 partition, default topic size, 0
> replication).
>
> And here are my NICE to haves (eventually):
>  - You should have the ability to tune a kafka topic's config from the UI,
> such as setting the topic size, # of partitions, and replication number.
> This is especially useful if you just created the kafka topic in the UI.
> Of course this should have lots of warnings about the issues with
> decreasing these numbers in the future, and maybe a "read more" link to
> some related documentation in the kafka documentation.
>
>  - Tune and restart the topologies to add parallelism into storm (things
> like the --parser_p, --parser_num_tasks, --num_workers, etc. options).
>  - It would be nice to be able to modify the jobs that purge data from HDFS
> and ES via the UI.  Managing the cluster should include setting your data
> retention wants - I think these are still purge cron jobs?  For some
> context, I'm thinking about how this could manage METRON-477 (both the
> original ticket as well as Nick's comments for additional features), which
> could be managed in this UI.
>  - The ability to generate a mpack that would install the currently
> configured state of the cluster (not including data, of course, but
> focusing on setting new, tweaked defaults to be what is currently
> configured).  This would be useful for rebuilds, standing up new clusters,
> or migrating to a different environment.  IMO, this is a really slick
> feature because it allows a nice migration from dev > test > prod.
>
>
> Not to get off topic, but I think the ES side of the house could generally
> use some love.  For instance, I don't see a reason why every string field
> shouldn't have '"ignore_above": 8191' on it, at least right now.  I will
> probably do this as a part of METRON-635, but we should make sure to carry
> that setting forward appropriately (i.e. if we infer ES templates from a
> schema).  I could agree with ES templates being inferred when creating new
> parsers/sensors as a nice to have, building off of a must have of being
> able to make/tweak ES templates in the UI/API.
>
>
> Ok, that's a lot of text, I'll end it there.  =D
>
>
> Jon
>
> On Wed, Feb 1, 2017 at 3:07 PM Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
> The MPack creates elastic templates for the standard parsers included in
> the pack.
>
> However, when creating new sensors we should be able to infer Elastic
> templates from the schema. There are certain fields that we need to mark
> types for to generate those templates (e.g. date / time fields). This goes
> back to the debate we had a while ago about, for example, typing in the
> grok parser, but applies for others too. I suspect that since we should be
> able to infer all the types needed for templates, this is more an excellent
> feature enhancement for the REST api layer, unless anyone can think of a
> reason we would need explicit type definition in the schema config element
> of the UI. Thoughts?
>
> Simon
>
>
>> On 1 Feb 2017, at 19:54, Nick Allen  wrote:
>>
>> The Index templates are currently handled in the Ambari MPack (unless you
>> are suggesting some additional functionality beyond what the Ambari MPack
>> gives us today.)
>>
>> I think there is always going to be a fuzzy, gray line between what the
>> Ambari MPack should do and what the Management UI should do.  I think it
>> works fine in Ambari today, but I could be convinced otherwise.
>>
>>
>>
>> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov 
>> wrote:
>>
>>> Thank you 

Re: [DISCUSS] Metron Management UI

[zeo...@gmail.com]

Ok great, thanks for the feedback Ryan.  I'm going to try and get around to
playing with this next week if I can.  Currently my productivity machine is
"in the shop" getting the battery replaced, so I'm playing man down right
now.

So if we ignore the current API restrictions, here are my MUST haves:

 - We should be able to update search/retrieve (i.e. I agree with Dima).
This means tweaking the ES templates, because when putting together a new
sensor or modifying an existing one, we will likely have a new field or
entirely new template.
 - When creating a new sensor, it should create a new kafka topic (if it
doesn't already - it may, I just haven't played with it yet) with some very
simple/dumb default settings (1 partition, default topic size, 0
replication).

And here are my NICE to haves (eventually):
 - You should have the ability to tune a kafka topic's config from the UI,
such as setting the topic size, # of partitions, and replication number.
This is especially useful if you just created the kafka topic in the UI.
Of course this should have lots of warnings about the issues with
decreasing these numbers in the future, and maybe a "read more" link to
some related documentation in the kafka documentation.

 - Tune and restart the topologies to add parallelism into storm (things
like the --parser_p, --parser_num_tasks, --num_workers, etc. options).
 - It would be nice to be able to modify the jobs that purge data from HDFS
and ES via the UI.  Managing the cluster should include setting your data
retention wants - I think these are still purge cron jobs?  For some
context, I'm thinking about how this could manage METRON-477 (both the
original ticket as well as Nick's comments for additional features), which
could be managed in this UI.
 - The ability to generate a mpack that would install the currently
configured state of the cluster (not including data, of course, but
focusing on setting new, tweaked defaults to be what is currently
configured).  This would be useful for rebuilds, standing up new clusters,
or migrating to a different environment.  IMO, this is a really slick
feature because it allows a nice migration from dev > test > prod.


Not to get off topic, but I think the ES side of the house could generally
use some love.  For instance, I don't see a reason why every string field
shouldn't have '"ignore_above": 8191' on it, at least right now.  I will
probably do this as a part of METRON-635, but we should make sure to carry
that setting forward appropriately (i.e. if we infer ES templates from a
schema).  I could agree with ES templates being inferred when creating new
parsers/sensors as a nice to have, building off of a must have of being
able to make/tweak ES templates in the UI/API.


Ok, that's a lot of text, I'll end it there.  =D


Jon

On Wed, Feb 1, 2017 at 3:07 PM Simon Elliston Ball <
si...@simonellistonball.com> wrote:

The MPack creates elastic templates for the standard parsers included in
the pack.

However, when creating new sensors we should be able to infer Elastic
templates from the schema. There are certain fields that we need to mark
types for to generate those templates (e.g. date / time fields). This goes
back to the debate we had a while ago about, for example, typing in the
grok parser, but applies for others too. I suspect that since we should be
able to infer all the types needed for templates, this is more an excellent
feature enhancement for the REST api layer, unless anyone can think of a
reason we would need explicit type definition in the schema config element
of the UI. Thoughts?

Simon


> On 1 Feb 2017, at 19:54, Nick Allen  wrote:
>
> The Index templates are currently handled in the Ambari MPack (unless you
> are suggesting some additional functionality beyond what the Ambari MPack
> gives us today.)
>
> I think there is always going to be a fuzzy, gray line between what the
> Ambari MPack should do and what the Management UI should do.  I think it
> works fine in Ambari today, but I could be convinced otherwise.
>
>
>
> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov 
> wrote:
>
>> Thank you Houshang,
>>
>> That looks absolutely great, loved it!
>> Will it make sense to also add ElasticSearch index  templates
>> querying/creation here?
>>
>> - Dima
>>
>> On 02/01/2017 02:06 AM, Houshang Livian wrote:
>>> Hello Metron Community,
>>>
>>> We have constructed a Management Module UI, built on top of METRON-503
>> (REST API) (currently under review). This Module gives users the ability
to
>> setup and administer much of the product through the UI.
>>>
>>> Here are some screens to show you what we are thinking. Please take a
>> look:
>>> http://imgur.com/a/QAQyu?
>>>
>>> Does this look like a reasonable place to start?
>>> Is there anything that is an absolute MUST have or MUST NOT have?
>>>
>>> Houshang Livian
>>> Senior User Experience Designer
>>> Hortonworks
>>>
>>> www.hortonworks.com

Re: [DISCUSS] Metron Management UI

[Simon Elliston Ball]

The MPack creates elastic templates for the standard parsers included in the 
pack. 

However, when creating new sensors we should be able to infer Elastic templates 
from the schema. There are certain fields that we need to mark types for to 
generate those templates (e.g. date / time fields). This goes back to the 
debate we had a while ago about, for example, typing in the grok parser, but 
applies for others too. I suspect that since we should be able to infer all the 
types needed for templates, this is more an excellent feature enhancement for 
the REST api layer, unless anyone can think of a reason we would need explicit 
type definition in the schema config element of the UI. Thoughts? 

Simon


> On 1 Feb 2017, at 19:54, Nick Allen  wrote:
> 
> The Index templates are currently handled in the Ambari MPack (unless you
> are suggesting some additional functionality beyond what the Ambari MPack
> gives us today.)
> 
> I think there is always going to be a fuzzy, gray line between what the
> Ambari MPack should do and what the Management UI should do.  I think it
> works fine in Ambari today, but I could be convinced otherwise.
> 
> 
> 
> On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov 
> wrote:
> 
>> Thank you Houshang,
>> 
>> That looks absolutely great, loved it!
>> Will it make sense to also add ElasticSearch index  templates
>> querying/creation here?
>> 
>> - Dima
>> 
>> On 02/01/2017 02:06 AM, Houshang Livian wrote:
>>> Hello Metron Community,
>>> 
>>> We have constructed a Management Module UI, built on top of METRON-503
>> (REST API) (currently under review). This Module gives users the ability to
>> setup and administer much of the product through the UI.
>>> 
>>> Here are some screens to show you what we are thinking. Please take a
>> look:
>>> http://imgur.com/a/QAQyu?
>>> 
>>> Does this look like a reasonable place to start?
>>> Is there anything that is an absolute MUST have or MUST NOT have?
>>> 
>>> Houshang Livian
>>> Senior User Experience Designer
>>> Hortonworks
>>> 
>>> www.hortonworks.com
>>> 
>>> 
>>> Mobile: (831) 521-4176
>>> hliv...@hortonworks.com
>>> 
>>> 
>>> 
>> 
>> 
> 
> 
> -- 
> Nick Allen 

Re: [DISCUSS] Metron Management UI

[Nick Allen]

The Index templates are currently handled in the Ambari MPack (unless you
are suggesting some additional functionality beyond what the Ambari MPack
gives us today.)

I think there is always going to be a fuzzy, gray line between what the
Ambari MPack should do and what the Management UI should do.  I think it
works fine in Ambari today, but I could be convinced otherwise.



On Wed, Feb 1, 2017 at 9:51 AM, Dima Kovalyov 
wrote:

> Thank you Houshang,
>
> That looks absolutely great, loved it!
> Will it make sense to also add ElasticSearch index  templates
> querying/creation here?
>
> - Dima
>
> On 02/01/2017 02:06 AM, Houshang Livian wrote:
> > Hello Metron Community,
> >
> > We have constructed a Management Module UI, built on top of METRON-503
> (REST API) (currently under review). This Module gives users the ability to
> setup and administer much of the product through the UI.
> >
> > Here are some screens to show you what we are thinking. Please take a
> look:
> > http://imgur.com/a/QAQyu?
> >
> > Does this look like a reasonable place to start?
> > Is there anything that is an absolute MUST have or MUST NOT have?
> >
> > Houshang Livian
> > Senior User Experience Designer
> > Hortonworks
> >
> > www.hortonworks.com
> > 
> >
> > Mobile: (831) 521-4176
> > hliv...@hortonworks.com
> > 
> >
> >
>
>


-- 
Nick Allen 

Re: [DISCUSS] Metron Management UI

[Houshang Livian]

Hello Dima,

Thank you for taking a few minutes to look at this new addition. I hope that it 
can make working with Metron much easier.

Do you think ElasticSearch Index Templates are a MUST have initially? Or is it 
okay to start here?

I do like the idea of adding that functionality.

Houshang




On 2/1/17, 6:51 AM, "Dima Kovalyov"  wrote:

>Thank you Houshang,
>
>That looks absolutely great, loved it!
>Will it make sense to also add ElasticSearch index  templates
>querying/creation here?
>
>- Dima 
>
>On 02/01/2017 02:06 AM, Houshang Livian wrote:
>> Hello Metron Community,
>>
>> We have constructed a Management Module UI, built on top of METRON-503 (REST 
>> API) (currently under review). This Module gives users the ability to setup 
>> and administer much of the product through the UI.
>>
>> Here are some screens to show you what we are thinking. Please take a look:
>> http://imgur.com/a/QAQyu?
>>
>> Does this look like a reasonable place to start?
>> Is there anything that is an absolute MUST have or MUST NOT have?
>>
>> Houshang Livian
>> Senior User Experience Designer
>> Hortonworks
>>
>> www.hortonworks.com
>> 
>>
>> Mobile: (831) 521-4176
>> hliv...@hortonworks.com
>> 
>>
>>
>
>

Re: [DISCUSS] Metron Management UI

[Kyle Richardson]

This looks awesome! A great starting point. Like Jon, I'm looking forward
to kicking the tires.

-Kyle

On Wed, Feb 1, 2017 at 10:18 AM, Ryan Merriman  wrote:

> Jon, I've done my best to keep this UI in sync with the API PR so you can
> play around with it now if you want.  There are 2 different versions of the
> UI:  one without the Blockly editor (
> https://github.com/merrimanr/incubator-metron/tree/METRON-623) and one
> with
> (https://github.com/merrimanr/incubator-metron/tree/blockly).  The Blockly
> editor feature contains a significant amount of code so we decide to keep
> it in it's own branch so reviewing will be easier.
>
> If you are familiar with the javascript ecosystem it's not hard to start it
> up but some instructions will definitely help.  I will add a README to make
> this easier but here's a quick summary:
>
> Start up the Docker environment in metron-docker
> Set the "docker.host.address" property in
> /incubator-metron/metron-interface/metron-rest/src/
> main/resources/application-docker.yml
> to match the IP address of your Docker machine
> Run org.apache.metron.rest.MetronRestApplication with spring active
> profiles set to "docker,dev"
> Navigate to metron-interface/metron-config and run "npm install"
> Start the UI with "metron-config/scripts/start-dev.sh"
>
> We're still working on a seamless install process but this should work in
> the meantime.  Feel free to reach out if you need help getting it going.
> As for feedback, I wouldn't limit it to what the API currently provides the
> API PR.  We can add services as needed.
>
> Ryan
>
> On Tue, Jan 31, 2017 at 7:49 PM, zeo...@gmail.com 
> wrote:
>
> > First off - this is an awesome first take at a management UI and I'm
> > looking forward to messing around with it.  Other than skimming some of
> the
> > dialogue as it comes in I have not been keeping up with the API PR.
> Should
> > I be able to assume that the UI PR is broken until the API is merged and
> > the UI can be updated for any changes?  I have not spun up the UI based
> on
> > that assumption.
> >
> > Are you looking for feedback that's limited to what can currently be done
> > using the API (a CRUD interface for SensorParserConfigs)?
> >
> > Thanks,
> >
> > Jon
> >
> > On Tue, Jan 31, 2017, 7:06 PM Houshang Livian 
> > wrote:
> >
> > Hello Metron Community,
> >
> > We have constructed a Management Module UI, built on top of METRON-503
> > (REST API) (currently under review). This Module gives users the ability
> to
> > setup and administer much of the product through the UI.
> >
> > Here are some screens to show you what we are thinking. Please take a
> look:
> > http://imgur.com/a/QAQyu?
> >
> > Does this look like a reasonable place to start?
> > Is there anything that is an absolute MUST have or MUST NOT have?
> >
> > Houshang Livian
> > Senior User Experience Designer
> > Hortonworks
> >
> > www.hortonworks.com
> > 
> >
> > Mobile: (831) 521-4176
> > hliv...@hortonworks.com
> > 
> >
> > --
> >
> > Jon
> >
> > Sent from my mobile device
> >
>

Re: [DISCUSS] Metron Management UI

[Dima Kovalyov]

Thank you Houshang,

That looks absolutely great, loved it!
Will it make sense to also add ElasticSearch index  templates
querying/creation here?

- Dima 

On 02/01/2017 02:06 AM, Houshang Livian wrote:
> Hello Metron Community,
>
> We have constructed a Management Module UI, built on top of METRON-503 (REST 
> API) (currently under review). This Module gives users the ability to setup 
> and administer much of the product through the UI.
>
> Here are some screens to show you what we are thinking. Please take a look:
> http://imgur.com/a/QAQyu?
>
> Does this look like a reasonable place to start?
> Is there anything that is an absolute MUST have or MUST NOT have?
>
> Houshang Livian
> Senior User Experience Designer
> Hortonworks
>
> www.hortonworks.com
> 
>
> Mobile: (831) 521-4176
> hliv...@hortonworks.com
> 
>
>

Re: [DISCUSS] Metron Management UI

[zeo...@gmail.com]

First off - this is an awesome first take at a management UI and I'm
looking forward to messing around with it.  Other than skimming some of the
dialogue as it comes in I have not been keeping up with the API PR.  Should
I be able to assume that the UI PR is broken until the API is merged and
the UI can be updated for any changes?  I have not spun up the UI based on
that assumption.

Are you looking for feedback that's limited to what can currently be done
using the API (a CRUD interface for SensorParserConfigs)?

Thanks,

Jon

On Tue, Jan 31, 2017, 7:06 PM Houshang Livian 
wrote:

Hello Metron Community,

We have constructed a Management Module UI, built on top of METRON-503
(REST API) (currently under review). This Module gives users the ability to
setup and administer much of the product through the UI.

Here are some screens to show you what we are thinking. Please take a look:
http://imgur.com/a/QAQyu?

Does this look like a reasonable place to start?
Is there anything that is an absolute MUST have or MUST NOT have?

Houshang Livian
Senior User Experience Designer
Hortonworks

www.hortonworks.com


Mobile: (831) 521-4176
hliv...@hortonworks.com


-- 

Jon

Sent from my mobile device

[DISCUSS] Metron Management UI

[Houshang Livian]

Hello Metron Community,

We have constructed a Management Module UI, built on top of METRON-503 (REST 
API) (currently under review). This Module gives users the ability to setup and 
administer much of the product through the UI.

Here are some screens to show you what we are thinking. Please take a look:
http://imgur.com/a/QAQyu?

Does this look like a reasonable place to start?
Is there anything that is an absolute MUST have or MUST NOT have?

Houshang Livian
Senior User Experience Designer
Hortonworks

www.hortonworks.com


Mobile: (831) 521-4176
hliv...@hortonworks.com